Blue Green Deployment with high volume databases

I'm working on changing the way we release new versions of of web software, and I have based it on the Blue Green Deployment model as blogged by Martin Fowler about the continuous delivery book. This approach is sometimes called "Flip-Flop" deployment.

This model works very well, but there is one key issue with it: How do you cope with databases that have a large volume of writes (non-interuptable) when you want to make significant schema changes?

So for example you have a system that is continously writing events to a database, and you want to change that schema in a release. In this case you would release to the blue slice and copy the data from the green database into (with a migration script), but then whilst you are testing the blue slice more data is being written to the green database. So when you make the blue slice live it won't have all the data - no problem, just copy the data again.

But what happens if you want to revert back to the green database after being live for a short while? for example in the case where you find that the release was bad - you can't just copy the data from the blue database (after all the release was bad so the data may be corrupted) and its in a different format (you could of course write another migration script to allow reverting back).

Another model is to put something in between the database and the data access layer:

This is what I'm viewing as being a database T splitter - you can ask your database layer to write to two databases (where the split occurs really comes down to how you have designed your database layer). This way you can turn the split on, and data will be written to both databases - thus allowing you at anypoint to have two incompatible schemas used. Of course if you design your data access using a pipe & filter pattern this would be very easy.

Another way around this issue would be to use a queueing mechanism, with some kind of replay ability. During a deployment the queue can be asked to remember all messages, and in the case of a bad release you can replay these messages.

The key point here is that blue-green does not magically fix issues with hihg-volume database systems - you will probably have to make some architectural changes in your system to support it.

Development's Naive Cynicism

Development is the process of creating a new product or release. In any development team or group there is a generally an accepted "right way" to develop a product (especially in the pathological team of one). However what I've observed in practice is that individuals in the team rarely follow the "right way” and bizarrely act (or report) as if the team is – I’m naming this behaviour as “Development's Naive Cynicism” (DNC)

Being cynical about the "right way"
We are cynical as we know what the "right way" is, but chose not to follow it. "Our process says do X and Y, but no-one does that!" is a common line. I'm not suggesting what the right way is or that there is only one "right way". The definition of "right way" changes dependent on approach (agile, lean and waterfall etc), constraints (time, money and skills etc) and suitability (complexity, duration and urgency etc) etc. Also two teams can have very different views of what the "right way" is, however within a team an agreement of "right way" is possible and desirable.

Being naive
We are naive as know that everyone is cynically not following the "right way", but chose to behave as though we do not know this. This becomes extreme in the case of a team of one, where we default to a sub-optimal approach as the "right way" is too much effort. Given that the definition of "right way" is under our own control (as we define it) this behaviour is bizarre - we chose not to redefine the "right way" (as we secretly know it IS the right way), but instead betray ourselves.

Why does Development's Naive Cynicism exist?
Let us recap: Development's Naive Cynicism = we know the right way, we don’t follow it, but pretend we are following it. What would cause us to exhibit this behaviour?

• Incorrect definition: the "right way" is not the right way, at some level we don't really believe in our definition of "right way". But do we ever make the effort to fix/redefine it?
• Self analysis: introspection is hard, and generalising our behaviour difficult. But why spend so much time learning if we don't seek to improve our core (kernel process) competence.
• Time: I've got a product/release to ship, get out of the way! But surely (and don't call me Shirley) we know that we would be faster, quicker and lighter etc if we did it the “right way”.
• Laziness: it's too much effort, no one else is doing either! Not really going to be a professional with this line of thinking

I think one of the main reason that DNC continues is the effort/reward perception, and durability of desire - only a slow steady improvement in process seems to work (big bang changes fail), which yields slow rewards, factored in the difficulty in persisting with continuous improvement makes it a difficult for an individual let alone a collective to eradicate it. A secondary issue is complexity, as technology progresses it tends to create complexity, however there is not a matching effort put in to change processes to cope with this - so overtime all processes become unfit for purpose.

That leaves us with two choices:

• Stop DNC: either by living up to the promises made by our "right way", or redefine our "right way" to reflect what we actually do
• Continue with DNC: maybe it does not exists, maybe it’s not that important or maybe it’s too painful to face into?

Requirements: Non-directive interviews

Reading the Economist's article "The avatar will see you now" I came across:

"Meanwhile, a real person alongside the participant will answer any questions and, after the virtual visit is over, Dr Conboy-Hill will carry out “non-directive” interviews, a technique used by the police to encourage witnesses to choose what they want to talk about and to maximise recall"

What interested me was the snippet about "non-directive" interviews - it sounded like a technique that might be worthwhile for requirements gathering - I subsequently found "ACTIVE LISTENING IN THE NONDIRECTIVE INTERVIEW" which I think there are a few tricks worth picking up - especially rephrasing and the "brake shoes"...

of course I'm a sucker for any document that uses the word "interlocutor" ;-)