Archive for category inkzee
Cherokee Web Server… when lighter and faster is a must
Sometimes when it comes to choosing a web server we always end up in the same decision: Apache… it’s alright, it’s the most widely used web server but that doesn’t mean it’s the best one.
When we started with Inkzee we didn’t think too much about other alternatives… we started with Apache and it’s not that we had big complaints about it, but having all the backend in one machine made us have performance/low resource consumption requirements and after a few months of our service running the all the different backend subsystems were almost killing the server and we started to research alternatives.
We were going to change Apache for Lighttpd but then we stepped into Cherokee statistics and after reading some reviews we decided to give it a try, the numbers spoke for themselves. We had been checking on Che
rokee for some years now, the project was mature enough for our needs so we went for it.
After a year and a half we can write our own lessons-learned from our experience:
- The Cherokee team have been so responsive to every problem or tweak we wanted to do with our server.
- The admin interface is pretty rad, if you are not used to messing with config files you’ll find no challenge.
- We didn’t miss anything: FastCGI, PHP, TLS and SSL, Load Balancer, Reverse Proxy, statics cache, …
- And the most important for us: our servers stand the load so much better now… indeed now we are still able to have our service in just one server
So, dare to try it now that Cherokee is celebrating their version 1.0 release, it’s a really nice product worth trying.
Using Zend Framework
One of our team members is a testing ninja and we bring him on board exactly because of that. We wanted to make our testing framework more robust so we asked for his wise expertise. One of the things he proposed was the use of Zend Framework. I’m not very keen of frameworks but it’s true that from a code point of view, we do need to get a better MVC architecture than the one we have now.
This is specially true now that it’s not just me coding, but a couple more people, so we are migrating part of the code to the Zend Framework or more precisely, to a tuned version of it we want to build. The idea is to only use what’s necessary to be able to create interface tests in a very reliable way. Right now this is not entirely possible without Selenium (our testing UI server), and what we want is to don’t need a webserver to run the tests.
We’ll be migrating parts of the UI and will be living with both environments until we’ve migrated all the code.
The corporate road
Since the beginning of this year, we are starting to focus heavily on the corporate side of our business. The reason? Money, of course. Although the reader is very useful, we feel it has an even greater potential if we tune it for the corporate world.
That’s the reason why since the beginning of the year we’ve been working on an enterprise product. It’s a simplified version of the reader, with new features, but very oriented to smart and seamless integration within a corporate web.
Check out this simple demo we recorded some time ago:
So basically it’s a way to create groups of feeds that generate a single url that can be used to subscribe to from a web widget or similar. Right now we are working on adding new features to this very simple version. We are also working with our first corporate customers to roll out the first pilots inside companies.
If you might be interested in testing it, let us know and we’ll get in contact with you!
The team grows
Hi everyone!
Sorry for the lack of communication this past months. I’ll try to write about everything that’s been going on recently with Inkzee.
One of the most important news is that the team is growing. We’ve incorporated new members to the Inkzee family. We are very happy because we have so many things to do at once that we definitely need more hands to help. It’s very hard to keep pushing on your own so I’m grateful of having people helping Inkzee apart from myself.
The interesting part is that things have change considerably. When on your own, you don’t need to explain processes, code or features. Now we need to document everything, from code structure to deploying methods and that takes time. We setup an internal wiki, moved the local trac to a server accessible to the other members and did a couple of training sessions with the new members to get them up to speed.
But we are finally there and starting to work as a team!!
The importance of resting
Happy New Year all!
During this Christmas, we gave ourselves a little break and I have to say, it felt really good! We entrepreneurs love to do things, we are generally over excited by everything and it’s rare if we aren’t handling several projects at the same time. The problem is that we don’t know when to take a break. Not resting, not knowing when to stop and watch movies with popcorn affects the performance of whatever we do.
Exactly that was what was happening me. My performance in the last weeks of December was terrible and I couldn’t concentrate in my coding. So, having been here before I immediately recognized it as what it was, lack of resting. So, I took 10 days of Christmas holidays.
Now I’m back, with a lot of energy for the new year a full of surprises for the next months! So, stay tuned for some news and remember, take a break from time to time, it’s good for your health
Efficent comunication protocols are critical
As we said before, one on the problems we had when migrating to AWS was that the backend system was putting a lot of stress on the server. After a week of benchmarking we realized that the protocol we where using to communicate between the different subsystems we have was the main responsible for the load increment.
From day one we didn’t want to use complex and cryptic protocols so we chose xmlrpc for our communication channel. It was easy to implement, had wide support in php and python and was very easy to debug. We knew that at some point we would need to switch to a more efficient protocol, but we didn’t know it was going to be so soon.
After doing some extensive benchmarking we realize that the through output of the protocol was very low, not only that, if too many xmlrpc connections were spawned it would eventually consume all resources of the process (file descriptors, sockets and memory). This was a painful lesson to learn, but we did. So we switched to the most efficient protocol we could found, that is binary. To be more precise we employ python’s cPickle binary protocol. Saying that the use of this is orders of magnitude more efficient is not even close 
So after switching each subsystem to the new protocol we saw the load of the machines going down. As with all big changes in the backend of any system, it took a while to stabilize it. To avoid any havoc we actually put it into production subsystem by subsystem so during some time we had both protocols running at the same time.
And so, always remember that the choices you make will come to hunt you if not done correctly
Merry Christmas 2009!
Hi all,
We just wanted to wish you all a Merry Christmas and a happy New Year!! We hope that this incoming new year is even better than the past one and we wish you all the best of lucks!
Turkey cheers from the Inkzee Team
Christmas updates
Hi all,
Sorry we’ve been so silent lately. It’s been a mad fall. Last post we wrote about a migration to Amazon Web Services (AWS). To be fair, the migration was a little painful. We had to set new machines with new software versions that made the system a little shaky for a while. We also run into some really nasty performance issues with the new AWS instances.
First of all, if you have a Debian image as we had, be sure to use the correct fine tuned C libraries for Xen Virtualization (that’s what AWS runs on). We started having a huge load on the machines and after investigating we discovered we were using the wrong C library and it was really killing the server.
So after changing the libraries and setting up images that would autoconfigure themselves with the latest config files and code from the latest release, we setup the monitoring software. That’s when we discovered the magic of the AWS instances. When you’re the only one running stuff in your datacenter, everything runs smoothly, but when the US East Coast woke up, the performance of the machine would just drop alarmingly. The reason is that, when no one else is using the resources of the machine your VM is running, you’ll get extra resources, but when all the resources are being used, you’ll get capped to what is configured for that AWS instance. In our case, that was killing us.
After figuring out this, we started benchmarking the backend and looking for ways to decrease the stress. When we finally did, everything went back to normal. We had to fine tune some parameters but the worse part was done. That was our great AWS adventure that didn’t end up there. In the next posts we’ll write a little bit about different issues we had and things we’ve been doing lately.
Step 3: AWS migration
Hello all!
We are finally there! We’ve devoted this past month to migrate all of the Inkzee system to the new database we are deploying, Tokyo Cabinet. It’s taken us forever to finish this migration but it’s here to stay. We are still missing some key things, but the bulk of it is already there so we’re starting to begin the migration to Amazon Web Services (AWS).
Right now we’ve managed to build a custom image for our servers with all the tools we need to run the Inkzee backend. Tomorrow and next week we’ll start moving data from the old server to the one in Amazon. This wont be painless and we know it. The architecture is still not 100% stable so we expect some minor glitches but nothing too serious.
So, here we go for step 3!!
Happy summer to all!
The Inkzee Team
Tokyo Tyrant and some numbers
Tokyo Tyrant is the database server that uses Tokyo Cabinet as backend. It allows you to access the database remotely. It supports 3 protocols, binary, memcache and http. This is great if you have already existing infrastructure.
We needed a php class that implemented the protocol so we took a look at two of them, Net_TokyoTyrant with Pete Warden’s patch and Tyrant by Bertrand Mansion. The first one supports http and binary protocols, while Tyrant only supports the raw binary protocol.
During the first tests, Net_TokyoTyrant went crazy when inserting over 28000 records over http, so I guess there’s something wrong with that. When we switched to the binary protocol it worked as expected.
Here are some quick numbers:
Net_TokyoTyrant (100000 keys)
Time inserted: 50.3662779331 secs
Time retrieved: 57.7555668354 secs
Time deleted: 34.1996 secs
Tyrant (100000 keys)
Time inserted: 39.330272913 secs
Time retrieved: 44.3433589935 secs
Time deleted: 26.9360201359 secs
The former is slightly faster so I guess we’ll go for it. Specially important is that the author keeps it up to date, which is also a plus!
The Inkzee Team
