(Crosspost on http://www.sk-gaming.com/blog/JFK/16317-Behind_the_scenes_of_wwwskgamingcom)
In my rare spare
time I am running the technical infrastructure of the largest eSports web page out
there. Since running approximately 4 months on a newly code page I wanted to take
the opportunity to get you up to date what basically „runs“ www.sk-gaming.com from a technical perspective and
what we did to improve the new page. I will go into very detail which might
only be interesting for readers with a technical background.
Virtualized
Our web
page runs solely on a bunch of Xen servers. Xen
is an open-source virtualization platform (Hypervisor) that allows us to
flexibly set up new virtual servers which can share the same physical hardware with
other virtual machines. With Xen we can separate services that should better be
on separate machines, utilize our physical hardware even better and move
servers from one physical machine to another in case of a hardware failure.
Stable
A long time
ago we’ve chosen Debian as our Linux
distribution because it has proven to be extraordinary stable and it provides a
very logical and lean approach to administrate it. Currently we are running the
AMD64 release of Debian etch, their current stable version, on both the Parent
Domain (dom0) of Xen and the
virtual machines (domU’s) to
leverage high amount of ram.
Scalable
To scale to
the increased demand we’re experiencing since running on the new page we set up
the load balancer keepalived to build
a redundant firewall (active/passive) which amongst other things distributes
the web requests to a farm of web servers. When developing the new page we took
care that the code can be spread among multiple web servers safely.
Light
More than a
year ago we switched our web server software from Apache to lighttpd
which I can say was one of the best decisions ever. Lighttpd, as the name says,
has a very low resource footprint, almost no memory usage and a really flexible
way of configuring it, without sacrificing any features we need to run www.sk-gaming.com. And best of all, it’s literally
fast as light.
Known
That we use
PHP as our scripting language is pretty much known and has not changed. Currently
we are running on PHP5 and use XCache
for caching PHP intermediate code. Interesting maybe for other XCache users is
that when we upgrade to the current xcache-1.2.2 we noticed a drop of around
30ms rendertime, even though XCache was already the fastest opcode cacher out there
(at least according to this benchmark).
Fast
On the old
page we used full page caching on some often loaded pages like the index page,
but it was implemented in a very shirt-sleeved way: Every minute a script
downloaded the site and pushed the content into the database. The site the user
received simply got that rendered site from the database and sent that to the
client. The problem with that implementation is obvious: it does not scale beyond
a few number of pages cached, needed special coding on these to work and still
put certain load on the database.
The
approach we needed was a more general one which would work on every page and
takes into account, that we have many pages which are user specific (e.g.
/member/JFK looks different for you than for you). The solution we now have is
leveraging the speed and the scalability that memcached offers , an in-memory cache for data objects, combined with some
code (which I might share in a future article) that does basically the
following:
- Check
if memcached has a cached version of the requested page
- If
yes: deliver it from there and stop processing
- If
no: go on
- Render
the page normally but put the output to a buffer
- Put
that buffer into memcached and deliver the rendered content
With that
and using memcached to cache some other functionalities that normally needs database
queries, we reduced the number of these queries to zero on cached pages and replaced
them with only four, always fast (because non-blocking and distributed),
memcached queries. What do zero database queries mean for the users? Page
rendering of cached pages only takes around 10ms, which is for most users
faster than the page actually needs to travel through the internet to your
computers.
Compliant
Most pages
are already XHTML 1.0 Transitional compliant which you can easily check with
the W3C
validator. We still have to work on some issues left, but we are committed
to make www.sk-gaming.com completely compliant
to current web standards.
Secure
To protect
your password from being stolen on logging in, we decided to submit your
passwords securely through an SSL encrypted connection. For extra security, we
added the “Lock” icon next to the form, where you can enter your username and
password, which leads you to a secure login form. This login form allows you to
enter your password in a secure environment and makes sure your password is
submitted a) only to us and b) only secured with HTTPS. When entering the username
and password on the (non-encrypted) front page that cannot be assured, because
the content could have been altered after leaving our web servers (man in the
middle attack). For that extra security to work it is important that these requirements
are fulfilled:
- You
receive no certificate errors when opening https://www.sk-gaming.com/login/
- You
have no rootkit or backdoor on your computer which captures key inputs or has
installed trusted root certificates
Future-proof
With this
solid basis we are looking into implementing and optimizing our infrastructure
even more. For example we are evaluating how our users can benefit from new
technologies like CardSpace/OpenID
or Semantic Web. I will
keep you updated.