Sorry we’ve been so silent lately. It’s been a mad fall. Last post we wrote about a migration to Amazon Web Services (AWS). To be fair, the migration was a little painful. We had to set new machines with new software versions that made the system a little shaky for a while. We also run into some really nasty performance issues with the new AWS instances.

First of all, if you have a Debian image as we had, be sure to use the correct fine tuned C libraries for Xen Virtualization (that’s what AWS runs on). We started having a huge load on the machines and after investigating we discovered we were using the wrong C library and it was really killing the server.

So after changing the libraries and setting up images that would autoconfigure themselves with the latest config files and code from the latest release, we setup the monitoring software. That’s when we discovered the magic of the AWS instances. When you’re the only one running stuff in your datacenter, everything runs smoothly, but when the US East Coast woke up, the performance of the machine would just drop alarmingly. The reason is that, when no one else is using the resources of the machine your VM is running, you’ll get extra resources, but when all the resources are being used, you’ll get capped to what is configured for that AWS instance. In our case, that was killing us.

After figuring out this, we started benchmarking the backend and looking for ways to decrease the stress. When we finally did, everything went back to normal. We had to fine tune some parameters but the worse part was done. That was our great AWS adventure that didn’t end up there. In the next posts we’ll write a little bit about different issues we had and things we’ve been doing lately.

We are finally there! We’ve devoted this past month to migrate all of the Inkzee system to the new database we are deploying, Tokyo Cabinet. It’s taken us forever to finish this migration but it’s here to stay. We are still missing some key things, but the bulk of it is already there so we’re starting to begin the migration to Amazon Web Services (AWS).

Right now we’ve managed to build a custom image for our servers with all the tools we need to run the Inkzee backend. Tomorrow and next week we’ll start moving data from the old server to the one in Amazon. This wont be painless and we know it. The architecture is still not 100% stable so we expect some minor glitches but nothing too serious.

So, here we go for step 3!!

Happy summer to all!

The Inkzee Team