We meditated the decision, and we got to the conclusion it was worth the effort and that we would, not only learn a lot during the process, but that we would get greater productivity on the long run. So off we went to implement the 5 steps Eric proposes:

1. Continuous integration server: We created our own home brewed mini system that allows us to add new tests that get started every time we need them. For now it’s a rather rudimentary system but does the job handsomely.
2. Source control commit check: We already had a source control system, so we just added all the commit checks. At first we thought this was a waste of time, but with some time we realized how many bugs we had/introduced without this. We added 3 very basic checks, a python syntax check (with pylint) , a very basic php syntax check and the automatic triggering of all the unit tests we have. These checks also enforced a common style rules for all the code under SVN, a somehow gruesome task at first (all the old code was triggering the syntax and style checks), but very rewarding in the end.
3. Simple deployment script: We had a very small deployment script but we fine tuned it so that it work flawlessly with the new AWS infrastructure. As for now, we make all the deployments in the same fashion, something that reduces the number of flaws you can introduce in this step. The script also creates a backup copy of the previous running code, in case you need to revert to the old version because of some critical error. As to date we’ve never needed to revert anything
4. Real-time alerting: We introduced a lot of Munin plugins to monitor all type of parameters on our servers. We even developed some very simple Munin plugins for Tokyo Cabinet. We are still missing some alerting here. We have all the graphs but we still need to setup some alerting framework to detect weird situations.
5. Root cause analysis (5 whys): This is probably the most critical part of the process. We’ve realized that this 5th step is what makes all the previous ones work. This process is an iterative one, you start with a small thing but with time, it will grow into an amazing process that’s is able to detect the slightness problem way before it makes it to the live servers. We’ve become used to always ask the 5 whys and it’s helping in improving the quality of the software we code.

So all in all, it’s an incredible experience we are still figuring out but that to date, has been impressive. We’ve managed to do 14 code uploads in a single day with no bugs whatsoever, plus stopped introducing small potentially critical bugs in the code base and all the way into production. If you want to give it a try, please do read the original posts by Eric Ries, you wont regret it.