From day one we didn’t want to use complex and cryptic protocols so we chose xmlrpc for our communication channel. It was easy to implement, had wide support in php and python and was very easy to debug. We knew that at some point we would need to switch to a more efficient protocol, but we didn’t know it was going to be so soon.

After doing some extensive benchmarking we realize that the through output of the protocol was very low, not only that, if too many xmlrpc connections were spawned it would eventually consume all resources of the process (file descriptors, sockets and memory). This was a painful lesson to learn, but we did. So we switched to the most efficient protocol we could found, that is binary. To be more precise we employ python’s cPickle binary protocol. Saying that the use of this is orders of magnitude more efficient is not even close

So after switching each subsystem to the new protocol we saw the load of the machines going down. As with all big changes in the backend of any system, it took a while to stabilize it. To avoid any havoc we actually put it into production subsystem by subsystem so during some time we had both protocols running at the same time.

And so, always remember that the choices you make will come to hunt you if not done correctly

Sorry we’ve been so silent lately. It’s been a mad fall. Last post we wrote about a migration to Amazon Web Services (AWS). To be fair, the migration was a little painful. We had to set new machines with new software versions that made the system a little shaky for a while. We also run into some really nasty performance issues with the new AWS instances.

First of all, if you have a Debian image as we had, be sure to use the correct fine tuned C libraries for Xen Virtualization (that’s what AWS runs on). We started having a huge load on the machines and after investigating we discovered we were using the wrong C library and it was really killing the server.

So after changing the libraries and setting up images that would autoconfigure themselves with the latest config files and code from the latest release, we setup the monitoring software. That’s when we discovered the magic of the AWS instances. When you’re the only one running stuff in your datacenter, everything runs smoothly, but when the US East Coast woke up, the performance of the machine would just drop alarmingly. The reason is that, when no one else is using the resources of the machine your VM is running, you’ll get extra resources, but when all the resources are being used, you’ll get capped to what is configured for that AWS instance. In our case, that was killing us.

After figuring out this, we started benchmarking the backend and looking for ways to decrease the stress. When we finally did, everything went back to normal. We had to fine tune some parameters but the worse part was done. That was our great AWS adventure that didn’t end up there. In the next posts we’ll write a little bit about different issues we had and things we’ve been doing lately.

We needed a php class that implemented the protocol so we took a look at two of them, Net_TokyoTyrant with Pete Warden’s patch and Tyrant by Bertrand Mansion. The first one supports http and binary protocols, while Tyrant only supports the raw binary protocol.

During the first tests, Net_TokyoTyrant went crazy when inserting over 28000 records over http, so I guess there’s something wrong with that. When we switched to the binary protocol it worked as expected.

Here are some quick numbers:

Net_TokyoTyrant (100000 keys)

Time inserted: 50.3662779331 secs
Time retrieved: 57.7555668354 secs
Time deleted: 34.1996 secs

Tyrant (100000 keys)

Time inserted: 39.330272913 secs
Time retrieved: 44.3433589935 secs
Time deleted: 26.9360201359 secs

The former is slightly faster so I guess we’ll go for it. Specially important is that the author keeps it up to date, which is also a plus!

The Inkzee Team

Some days ago we rolled out a new release. Apart from fixing a bunch of minor bugs, we implemented an internal improvement. One of the problems we were running into is the huge amount of posts the system is tracking (last week was something around 650.000 posts). Problem is that accessing a single table with that amount of records in it can be time expensive. The solution was simple, most of the posts that are stored are quite old (more than 3 or 4 months old) so we decided to move all those old posts to some other tables where they would be access much less. That meant that we offloaded a lot of information from the main post table, keeping the number of records there under control.

As usual, implementing this things isn’t as straight forward as we would like and we had to implement a specific logic so that if a user wanted to read more posts from a blog that had been archived we would return them to the user in a seamless way. It’s finally working and the site’s performance is considerably better, so we are quite happy.

During the next weeks we’ll be partitioning some tables so that the site goes even faster, we’ll keep you posted.

Thanks for your patient!

The Inkzee Team

Right now, you’ll be able to experience a much better performance when reading posts and fetching a list of titles from a given feed. During the next days we hope the response time will get better and better.

We’ve also migrated our webservers today from Apache to Cherokee. Although Apache is excellent in many ways, we don’t need a fully loaded Apache, so we’ve chosen Cherokee which is working fine for us.

Thanks a lot for your patient!! And please, do sends us an email with any feedback you’ve got!

Have a nice weekend all!