Blog of Robert Lacroix

If you watch the moves of Google for some time, you can recognize without being too paranoid a consistent strategy of collecting as much data about people as possible.

It began with search queries, which are a very good indication of what an indiviuum is interested in. But bringing the people to sign-up and log-in personally to your search engine makes, overcomes the quite inaccurate IP- or cookie-based method to get distinct users and gives a direct relation between queries and actual people sitting in front of the computer. How do you achieve that? The answer is simple: Offer a free eMail address which tons of space so that you never have to throw away a mail ever again (Google Mail) or a customizable start page (iGoogle) and take that log-in to the search page. Voilá, really personal search.

But that all limits Google to its own pages, wouldn't it be much better to get data about people, even when they are not visiting Google servers? Easy solution to that as well: Offer an affordable ad system (Google Ads) that is working fast, is easy to implement and gives webmasters a quick way to earn some money. Or set up a free, really cool web statistics tool (Google Analytics), that is superior to everything that is out there. Doing so, extends the "span of collection" significantly.

Overall this already produces a huge amount of "coverage": All Google websites, websites that want to make some money and websites that want good statistics. But let's not stop there: What about the location of people, wouldn't that be interesting, too? No sooner said than done: Implement a maps serivces (Google Maps) and make it available on cell phones, which, by nature, can be used to determine the location of the user very exact (in contrast to IP-based locating mechanisms). All cell phones can use the cell phone towers to determine their location quite good already, but putting GPS chips in them make the result very astonishing.

Nowhere near enough, after having the people's interests (search queries), their contacts and conversations (mail), visited webistes (search queries, ads, stats), location (maps), they just added a new interesting tool: Voice search (Google Mobile App), which can be used to extract even more information, for example gender or mood. That's possible because in contrast to traditional voice recognition systems on phones the recognition itself is not done on the cellphone, the recordings are sent to google and are processed there. And after having collected enough audio fragements of one's voice, it should be no problem to create vocal fingerprints and use it to recognize or imitate people (at least when you believe some hollywood movies).

My remarks draw a picture, that is most likely exaggerated (though it covers only a few Google Services), but it should be helpful to generate some awareness about the piles of data Google is accumulating.

When it comes to service monitoring, nagios is a pretty good tool, though it can only monitor services from outside. To add monitoring of metrics not exposed to the network, like disk usage, you need an agent running on the monitored server. nagios-statd, written as a python daemon, is such an agent.

From a management and security perspective it is good to keep the number of processes and open network ports as minimal as possible. Those that not only monitor service availability with nagios, but also resource usage with ganglia, already have the ganglia agent (gmond) running on each server. Adding nagios-statd would be the second agent that needs to be configured and kept up-to-date.

But what if you want to be alerted if one of the hard disks get full, load goes through the roof or the server begins using swap space? You would need nagios-statd although ganglia knows about everything. That's why I came up with the idea of using data from ganglia and monitor them with nagios and wrote a small php shellscript that accomplishes this task. There is for sure some work to do but I think it's a good start:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75

#!/usr/bin/php <?php ### Get command line arguments $host = $argv[1]; $metric = $argv[2]; $metric_unit = $argv[3]; $cluster_arg = $argv[4]; #optional $threshold_warn_arg = $argv[5]; #optional $threshold_crit_arg = $argv[6]; #optional ### Fill Variables if (!is_numeric($threshold_warn_arg))   $threshold_warn = 2; else   $threshold_warn = $threshold_warn_arg; if (!is_numeric($threshold_crit_arg))   $threshold_crit = 1; else   $threshold_crit = $threshold_crit_arg; if (!$cluster_arg)   $cluster = "sk"; else   $cluster = $cluster_args; ### Get data from gmond $fp = fsockopen("localhost", 8649, $errno, $errstr, 30); if (!$fp) {   echo "GANGLIA Unknown - $errstr ($errno)\n";   exit(3); } else {   while (!feof($fp)) {     $buffer .= fgets($fp, 128);   }   fclose($fp); } ### Get metric out of XML $xmlobj = simplexml_load_string($buffer); $metric_value = $xmlobj->xpath("/GANGLIA_XML/CLUSTER[@NAME='$cluster']/HOST[@NAME='$host']/ METRIC[@NAME='$metric']"); ### Convert data (more types tbd) $metric_value = $metric_value[0]->attributes(); if ($metric_value["TYPE"] == "double")   $metric_value = doubleval($metric_value["VAL"]); else   $metric_value = $metric_value["VAL"]; ### Build output strings $perfcounter = $metric . "=" . $metric_value . $metric_unit . ";" . $threshold_warn . $metric_unit . ";" . $threshold_crit . $metric_unit; $text = $metric . " is " . $metric_value . $metric_unit; ### Output if ($metric_value > $threshold_warn && $metric_value > $threshold_crit) {   print("GANGLIA OK - " . $text . " |" . $perfcounter . "\n");   exit(0); } if ($metric_value < $threshold_warn && $metric_value > $threshold_crit) {   print("GANGLIA Warning - " . $text . " |" . $perfcounter . "\n");   exit(1); } if ($metric_value < $threshold_warn && $metric_value < $threshold_crit) {   print("GANGLIA Critical - " . $text . " |" . $perfcounter . "\n");   exit(2); } echo "GANGLIA Unknown\n"; exit(3); ?>

Save the script in your nagios-plugin directory (Debian defaults to /usr/lib/nagios/plugins) and make it executable (chmod +x check_ganglia). You can then define a new command in your nagios config as follows (in this example disk_free):

1 2 3 4

define command{ command_name check_ganglia_disk_free command_line /usr/lib/nagios/plugins/check_ganglia $ARG1$ disk_free GB }

After that, add a new service to your host config:

1 2 3 4 5 6

define service{ use generic-service host_name xxx1 service_description DISK check_command check_ganglia_disk_free!xxx1 }

 Restart Nagios and you should be able to monitor disk usage from then on.

(Crosspost on http://www.sk-gaming.com/blog/JFK/16317-Behind_the_scenes_of_wwwskgamingcom)

In my rare spare time I am running the technical infrastructure of the largest eSports web page out there. Since running approximately 4 months on a newly code page I wanted to take the opportunity to get you up to date what basically „runs“ www.sk-gaming.com from a technical perspective and what we did to improve the new page. I will go into very detail which might only be interesting for readers with a technical background.

Virtualized

Our web page runs solely on a bunch of Xen servers. Xen is an open-source virtualization platform (Hypervisor) that allows us to flexibly set up new virtual servers which can share the same physical hardware with other virtual machines. With Xen we can separate services that should better be on separate machines, utilize our physical hardware even better and move servers from one physical machine to another in case of a hardware failure.

Stable

A long time ago we’ve chosen Debian as our Linux distribution because it has proven to be extraordinary stable and it provides a very logical and lean approach to administrate it. Currently we are running the AMD64 release of Debian etch, their current stable version, on both the Parent Domain (dom0) of Xen and the virtual machines (domU’s) to leverage high amount of ram.

Scalable

To scale to the increased demand we’re experiencing since running on the new page we set up the load balancer keepalived to build a redundant firewall (active/passive) which amongst other things distributes the web requests to a farm of web servers. When developing the new page we took care that the code can be spread among multiple web servers safely.

Light

More than a year ago we switched our web server software from Apache to lighttpd which I can say was one of the best decisions ever. Lighttpd, as the name says, has a very low resource footprint, almost no memory usage and a really flexible way of configuring it, without sacrificing any features we need to run www.sk-gaming.com. And best of all, it’s literally fast as light.

Known

That we use PHP as our scripting language is pretty much known and has not changed. Currently we are running on PHP5 and use XCache for caching PHP intermediate code. Interesting maybe for other XCache users is that when we upgrade to the current xcache-1.2.2 we noticed a drop of around 30ms rendertime, even though XCache was already the fastest opcode cacher out there (at least according to this benchmark).

Fast

On the old page we used full page caching on some often loaded pages like the index page, but it was implemented in a very shirt-sleeved way: Every minute a script downloaded the site and pushed the content into the database. The site the user received simply got that rendered site from the database and sent that to the client. The problem with that implementation is obvious: it does not scale beyond a few number of pages cached, needed special coding on these to work and still put certain load on the database.

The approach we needed was a more general one which would work on every page and takes into account, that we have many pages which are user specific (e.g. /member/JFK looks different for you than for you). The solution we now have is leveraging the speed and the scalability that memcached offers , an in-memory cache for data objects, combined with some code (which I might share in a future article) that does basically the following:

1. Check if memcached has a cached version of the requested page
2. If yes: deliver it from there and stop processing
3. If no: go on
4. Render the page normally but put the output to a buffer
5. Put that buffer into memcached and deliver the rendered content

With that and using memcached to cache some other functionalities that normally needs database queries, we reduced the number of these queries to zero on cached pages and replaced them with only four, always fast (because non-blocking and distributed), memcached queries. What do zero database queries mean for the users? Page rendering of cached pages only takes around 10ms, which is for most users faster than the page actually needs to travel through the internet to your computers.

Compliant

Most pages are already XHTML 1.0 Transitional compliant which you can easily check with the W3C validator. We still have to work on some issues left, but we are committed to make www.sk-gaming.com completely compliant to current web standards.

Secure

To protect your password from being stolen on logging in, we decided to submit your passwords securely through an SSL encrypted connection. For extra security, we added the “Lock” icon next to the form, where you can enter your username and password, which leads you to a secure login form. This login form allows you to enter your password in a secure environment and makes sure your password is submitted a) only to us and b) only secured with HTTPS. When entering the username and password on the (non-encrypted) front page that cannot be assured, because the content could have been altered after leaving our web servers (man in the middle attack). For that extra security to work it is important that these requirements are fulfilled:

1. You receive no certificate errors when opening https://www.sk-gaming.com/login/
2. You have no rootkit or backdoor on your computer which captures key inputs or has installed trusted root certificates

Future-proof

With this solid basis we are looking into implementing and optimizing our infrastructure even more. For example we are evaluating how our users can benefit from new technologies like CardSpace/OpenID or Semantic Web. I will keep you updated.

There are two ways of using the T-Mobile Multimedia Net Card on Mac. You can either download (and pay for) Mobile High Speed 3G which requries you to reboot during the installation or you use my modem script.
You need to download the modem script to the "Modem Scripts" folder first and assign the proper permissions. To do this paste the following into Terminal.app:

cd /Library/Modem\ Scripts/
sudo curl -o "T-Mobile Cellular" http://www.robertlacroix.com/files/T-Mobile%20Cellular
sudo chmod 664 T-Mobile\ Cellular

1. Disable PIN request on your T-Mobile GSM card, you won't be able to enter it :)
2. Plug in the Multimedia Net Card
3. Go to the Network Preference Pane
4. Choose PC Card Serial in the Show dropdown
5. Use the following settings in the PPP tab:
  Account Name: t-mobile
  Password: t-mobile
  Telephone Number: *99#
6. Choose Modem T-Mobile Cellular

Wait till the LED on the Net Card flashes blue for UMTS or green for GPRS and connect :)

When you set up Address Book to synchronize your contacts with Exchange hourly you probably know the "iSync couldn't connect to Exchange - Could not reach the server specified." for example when you are using your powerbook on the way.
I put a small shell script together which replaces the ExchangeScheduler binary that is called by Apple's scheduler hourly. It has a list of all network locations it should sync in, checks reachability of the outlook web access server and starts the ExchangeScheduler binary then.

Save the following as ExchangeScheduler.sh in
/System/Library/SyncServices/ExchangeConduit.bundle/Contents/Resources/

#!/bin/bash

syncLocations='73A6408D-CE03-11D8-B5C8-000A95D370A4 26415746-FF16-11D8-B1C7-000A95D370A4'
#syncLocations is a space seperated list of network location id's, the script should sync in.
#You can get your id's by entering scselect into Terminal.app.

ExchangeScheduler='/System/Library/SyncServices/ExchangeConduit.bundle/Contents/Resources/ExchangeScheduler.bin'
activeLocation=`scselect 2>&1 | egrep '^\ *\*'| cut -c 4- | cut -c -36`
for i in $syncLocations
do
	if [ "$activeLocation" == "$i" ]
	then
		if curl -k `defaults read com.apple.IISSupport IISServerPath` >/dev/null 2>&1
		then
			$ExchangeScheduler $*
		fi
	fi
done

Open Terminal.app and do the following commands

#change current directory:
cd /System/Library/SyncServices/ExchangeConduit.bundle/Contents/Resources/
#rename ExchangeScheduler to ExchangeScheduler.bin:
sudo mv ExchangeScheduler ExchangeScheduler.bin
#create symlink from ExchangeScheduler.sh to ExchangeScheduler:
sudo ln -s ExchangeScheduler.sh ExchangeScheduler
#grant execute permission to ExchangeScheduler.sh
sudo chmod +x ExchangeScheduler.sh

Test the script by activating a network location that is in $syncLocations and do

./ExchangeScheduler

If the Outlook Web Access Server is reachable the output should be

2004-09-05 10:11:51.302 ExchangeScheduler.bin[14242] Starting exchange sync

Finally the Mozilla guys fixed the hidden window on Mac OS X 10.3 which appeared when activating the Exposé functionality.
I made a build (hope it works for you) from the latest CVS checkout with the middle-mouse patch applied, which fixes middle-mouse click behavior on links to open a new tab. So actually two nasty bugs in Firefox on Mac OS X less :)