Posts Tagged scalability
When partitioning isn’t enough
These past weeks we’ve been partitioning our database design. The goal was to achieve better scalability. Because Inkzee grows with the number of feeds it holds, not the users, we needed to partitioned the data tables so that we could process feed posts faster.
After altering a lot of our current code so that it worked with the new database design we’ve been experiencing problems with MySQL. It seems that, even though the solution makes the overall system much faster (like 3 to 4 times faster), some operations don’t play too well with MySQL and add an unaccepted latency to the system.
We’ve been resisting the urge to migrate to a schema-less database but it seems we have no other option but to transition to it. So, even though we thought we could have the new design working by the end of the week, we are afraid we’ll have to postpone it until further notice. We’ll keep you guys updated though!
The Inkzee Team
Archiving news, what a wonderful thing
Hi all,
Some days ago we rolled out a new release. Apart from fixing a bunch of minor bugs, we implemented an internal improvement. One of the problems we were running into is the huge amount of posts the system is tracking (last week was something around 650.000 posts). Problem is that accessing a single table with that amount of records in it can be time expensive. The solution was simple, most of the posts that are stored are quite old (more than 3 or 4 months old) so we decided to move all those old posts to some other tables where they would be access much less. That meant that we offloaded a lot of information from the main post table, keeping the number of records there under control.
As usual, implementing this things isn’t as straight forward as we would like and we had to implement a specific logic so that if a user wanted to read more posts from a blog that had been archived we would return them to the user in a seamless way. It’s finally working and the site’s performance is considerably better, so we are quite happy.
During the next weeks we’ll be partitioning some tables so that the site goes even faster, we’ll keep you posted.
Thanks for your patient!
The Inkzee Team
The not so simple case of folder management
Today I want to tell all our readers a little tale, the case of the folder management. Folders has been one of the most asked features we got asked for in our first round of alpha testing. Users wanted to be able to group their feeds into folders. We started an early development and suddenly realized doing it wasn’t as simple as expected.
Technology seems easy to everyone except for the tech guys. Features that seem very straight forward become real nightmares to develop. The folder management was one of those. There where 2 steps we needed to take to implement it. The first one was to add all the necessary code in the user interface that enabled the user to drag and drop feeds into folders and within folders. That part was kind of easy, just some Javascript code here and there.
The second part wasn’t easy at all. We needed a way to store all the positions of all feeds and folders for a given user at any time. The easy way was to create a list to store the feed with the user’s position. Problem was that we needed to be able to reorder that list. That meant we needed to be able to change all the feeds ranking when a feed was reordered. Doing that translated into a lot of write operations to that list (and in conclusion to the servers disk). To those that aren’t tech geeks, write operations to any hard drive are very expensive, they take a long time and a lot of resources. Though if you want your servers to keep up with a big load of web traffic you need to minimize the number of write operations.
So we had this problem, if we had to change the ranking of all feeds every time some user reorder one of them, it meant a lot of write operations. The best case scenario was that the user dropped a feed at the end of this list, and so, we only needed to update this feeds rank. The worst case scenario was having the user drop a feed at the beginning of the list. In that case we needed to update all the other users feeds rank. When this list has only 10 feeds, then it’s ok, but if the list is as long as 800 feeds, those are a lot of write operations. Finally, if you start thinking we could have several users dragging and dropping feeds simultaneously, the number of possible write operations in the worst case was alarmingly high.
To solve the problem we partitioned the rank space so that ranks had a gap of x numbers in between. That allowed us to minimized the number of ranks we needed to update at a single moment. This added an extra layer of difficulty as we needed to track the number of free slots available before we needed to reorder the whole feed list.
At the same time we needed to track if a feed was being dragged, before or after another feed, before or after a folder or if it was being dropped inside a folder. So we had to check the type of operation, where, which folder and if we had free slots there.
When we finally had all this stuff right we had to translate this internal structure into the export / import mechanism that produces or reads opml files (xml files). We needed to be able to read an opml file with folders and translate it into our internal representation and ranking list so it would be correctly displayed to the users.
All in all, it was fun but a very hard work for something that initially seemed so trivial to accomplish. The system is working now and apart from eventual glitches it should work. We still want to get a better UI for the drag & drop thing as it seems to us it’s still a little bit clumsy, but we’ll get there.
So now you know that even the simplest feature requires a hard work, so choose wisely what you want us to develop next 
Architectural changes
Summer has been a slow month but we’ve been working on a new architecture for Inkzee. Right now, and even though we enabled new internal features, the current server where the alpha is hosted is too slow for the application and so we are migrating it to a bigger server.
Before we complete the migration, we are finishing some big changes to the way the application works. The new changes will make the application much faster and much more scalable. That way, if the new server isn’t enough, we’ll be able to add new machines to the application and scale horizontally.
The architectural changes will be finished by next week, so the migration will happen very soon and hopefully we’ll be able to send more invitations for the current alpha.
Thanks a lot for the comprehension and we hope everyone had a good summer! 
The Inkzee team
