From day one we didn’t want to use complex and cryptic protocols so we chose xmlrpc for our communication channel. It was easy to implement, had wide support in php and python and was very easy to debug. We knew that at some point we would need to switch to a more efficient protocol, but we didn’t know it was going to be so soon.

After doing some extensive benchmarking we realize that the through output of the protocol was very low, not only that, if too many xmlrpc connections were spawned it would eventually consume all resources of the process (file descriptors, sockets and memory). This was a painful lesson to learn, but we did. So we switched to the most efficient protocol we could found, that is binary. To be more precise we employ python’s cPickle binary protocol. Saying that the use of this is orders of magnitude more efficient is not even close

So after switching each subsystem to the new protocol we saw the load of the machines going down. As with all big changes in the backend of any system, it took a while to stabilize it. To avoid any havoc we actually put it into production subsystem by subsystem so during some time we had both protocols running at the same time.

And so, always remember that the choices you make will come to hunt you if not done correctly

After some finally tweaks, we managed to push another release today and finished migrating all users from the old alpha architecture. We were so excited, we decided it was time to send some more invites to some more users that have been waiting for it for ages! (We really regret that, sorry guys!).

Suddenly what we can only define as a shit storm came our way. Servers were on fire, even though we didn’t send too many invites. Problem is that our system scales with the number of feeds each users has. Some of our new alpha users are really feed junkies (you know who you are! ) and really pushed the total number of feeds the system manages really fast. We are throwing some stats so you get some idea:

As you can see, the number of feeds being tracked by Inkzee internally nearly triplicated in a 20 minute span. We are storing, displaying and sorting 32,859 posts right now and growing fast.

In the process of managing such a large amount of new feeds some parts of the backend went belly up. We are investigating it now and we hope we can fix them soon. Even though some things broke, they weren’t critical parts so the system recovered from the failures pretty fast which is good news.

The moment we stabilize the system again we’ll continue sending new invites. Right now we are also focusing in adding new features. The next one in our roadmap will be the ability of creating custom filters and keyword alerts. We hope to release that some time next week.

As usual, thanks a lot for your patient and keep coming, but gently

The Inkzee Team