Friday, June 4, 2010

Blue Green Deployment with high volume databases

I'm working on changing the way we release new versions of of web software, and I have based it on the Blue Green Deployment model as blogged by Martin Fowler about the continuous delivery book. This approach is sometimes called "Flip-Flop" deployment.

This model works very well, but there is one key issue with it: How do you cope with databases that have a large volume of writes (non-interuptable) when you want to make significant schema changes?
So for example you have a system that is continously writing events to a database, and you want to change that schema in a release. In this case you would release to the blue slice and copy the data from the green database into (with a migration script), but then whilst you are testing the blue slice more data is being written to the green database. So when you make the blue slice live it won't have all the data - no problem, just copy the data again.

But what happens if you want to revert back to the green database after being live for a short while? for example in the case where you find that the release was bad - you can't just copy the data from the blue database (after all the release was bad so the data may be corrupted) and its in a different format (you could of course write another migration script to allow reverting back).

Another model is to put something in between the database and the data access layer:

This is what I'm viewing as being a database T splitter - you can ask your database layer to write to two databases (where the split occurs really comes down to how you have designed your database layer). This way you can turn the split on, and data will be written to both databases - thus allowing you at anypoint to have two incompatible schemas used. Of course if you design your data access using a pipe & filter pattern this would be very easy.

Another way around this issue would be to use a queueing mechanism, with some kind of replay ability. During a deployment the queue can be asked to remember all messages, and in the case of a bad release you can replay these messages.

The key point here is that blue-green does not magically fix issues with hihg-volume database systems - you will probably have to make some architectural changes in your system to support it.

No comments: