Who are you ? The story about DBus and cloning virtual machines

Kronometrix Data Recording Module, ships with a transport utility called sender, which ensures all raw data is shipped to one or many data analytic appliances over HTTP or HTTPS protocols. 

sender, a Perl5 citizen, checks all raw data updates and relies on a host uuid identifier from the operating system to deliver that raw data. If such host uuid is found will use that for the entire duration of its execution. If no such host uuid is found it will generate a new one and store that on its configuration file, kronometrix.json. The analytics appliance, relies that each data source is unique and valid, properly checked or generated by the transport utility, sender. 

But what's happening if this really does not work ?


Data Source Id

Kronometrix uses the data source id concept, (DSID), to identify a data source point, from where one or many data messages are received. For example for IT, Computer Performance data, a DSID identifies to a computer system host, physical or virtual, connected or not to a TCP/IP network.

The DSID is obtained from operating system core functions. Example:
  • Linux platforms we speak to DBus and we try to get that via machine-id file
  • FreeBSD we ask the sysctl interface for kern.hostuuid
  • other way: we compute one, using UUID::Tiny Perl5 module


Who are you ?

Working closely with one of our customers, we've seen that they were not receiving data from a number of virtual machines where previously we have installed Kronometrix. Looking into this we discovered that sender was producing same data source id across a number of virtual machines, using same DSID:


"dsid" : "96d5b4a4-d0fa-54a8-ba74-14cc978041f1"







So to Kronometrix Analytics Appliance all these hosts were more or less similar, having same DSID. Not good.



















Whats wrong ?



As simple as that, we found out that one VM was used to clone other VMs and by mistake the machine-id file was cloned as well. DBus on CentOS 6.x did produce a sane and valid machine-id, but then the VM configuration was taken and cloned, including the machine-id. On this respect from operating system level all hosts were similar identified as having same host UUID. No software was reporting this nor complain about this malfunction.







Our system was able to immediately discover this trouble and we proposed a fix to the Operation Center group. Later the cloning procedure was fixed to ensure machine-id on Linux will not be cloned anymore and data was finally flying to our appliance. Nice and easy. 







No more clones :)




































Comments











Post a Comment



















Popular posts from this blog









Asus Zenbook and FreeBSD 11 




















Image







This is a short description how I got running FreeBSD 11-current on my Asus Zenbook UX32VD laptop. Im very happy with the current setup but of course it is room for improvements in many areas. Having DTrace, ZFS and the other goodies makes FreeBSD a real good candidate for a mobile environment.   Last Updated: 16 March 2015    Zenbook UX32VD Configuration   CPU: Intel(R) Core(TM) i7-3517U CPU @ 1.90GHz  Memory: 6 GB RAM  Storage: SanDisk SSD i100 24GB, Samsung SSD 840 PRO 500GB  Video: Intel HD Graphics 4000, Nvidia GT620M  Display: 13" IPS Full HD anti-glare  Wifi: Intel Centrino Advanced-N 6235  3 x USB 3.0 port(s)   1 x HDMI   1 x Mini VGA Port   1 x SD card reader  Note: The laptop's internal hard drive, has been replaced by a Samsung SSD.   








Read more










Raspberry Pi and Redis




















Image







This summer we did something amazing: we took our enterprise appliance, a very powerful server and we tried to run same software, same analytic kernel and all the other things in something smaller and lot simpler, like this:     Light, mobile appliance           And we did it ! Being powered by OpenResty and Lua we were able to size our appliance easily and be up and running quickly. Low factor, low power consumption, very cheap, these ARM based devices are getting more and more attention and become more popular nowadays:   Banana Pi - A Highend Single-Board Computer  4 x Raspberry Pi + Arduino = UDOO          What ? Analytics on a Raspberry PI ... !?   Yep. First of all, we wanted to experiment with ARM based boards using few number of hosts and few data messages per host as input. The immediate trouble was that we should fit in 512MB RAM with all our software and customer data. Then these boards don't run on latest Intel specs but something like 700 Mhz CPU as a system on chip wi








Read more










Web Performance, tell me the story




















Image







    Suppose you would like to know how your web site, behaves to clients from two different geo locations: for example, Finland and Ireland regarding response time. How quickly can you see and understand visually what's going on and how things are.      Do you need to improve something ? Change your network provider ? Tune your application(s) ? Spend more money.  Start from a top level view, understanding the big picture and having the possibility to dive and analyze at the detail level.  Finally, can you visually tell the story of your web application as in a short chart ?   Enter Kronometrix Web Performance ...              Finland      Something happened between 1:00 - 1:15 AM    We can see more than one request affected by this event    Overall we see all requests are executing fast, below 0.5 seconds    And we have some exceptions, some are taking more than 0.5 seconds             Ireland      Same things from Ireland    It is different, requests usually take longer to execute








Read more