Wond’ring aloud/how we feel/today
Photo courtesy Patryk Pigeon
From late 2005 on, there was a very interesting discussion about Web 2.0 and SOA. John Hagel, Nicholas Carr, Andrew McAfee and Dion Hinchcliffe were involved, amongst others. To refresh your memory (or to make it easier for you in the event you hadn’t actually come across the debate, here are some of the key links:
I was Global CIO at Dresdner Kleinwort at the time, and found the debate both timely as well as very relevant to the challenges we faced. Across the industry, the promise of high cohesion and loose coupling propounded by the web services revolution and SOA seemed to be somewhat remote, more standards-wars than design-principles in character; the expectation of a small-pieces-loosely-joined outcome seemed more and more unlikely to be met as a result, as work backlogs grew; those organisations that had implemented enterprise buses seemed to be affected less than those that hadn’t, but it still wasn’t pretty; everywhere we looked, there were variants of vertically integrated stacks, benighted in the belief that transaction costs would actually tumble as a result. While we were using a number of Web 2.0 technologies at the bank, they were not integrated with the transactional side of the bank, in terms of research and trading, and still some distance away from the back office operations.
It was around that time that we were learning more about how open multisided platforms could work, piggybacking on what the opensource community were doing, and, despite Stallman’s warnings to take care with the term, people started talking about software ecosystems. And that got me thinking more about the transaction costs aspect of these architectures.
Over time, what appeared to be happening was that SOA dominated the traditional “back office” and “transaction processing” worlds, while “Web 2.0″ approaches were used to deal with customer-facing applications. Now this was just anecdotal evidence, nothing deeply scientific about it…. but the schisms spoken of by Hagel and Carr and Hinchcliffe et al were becoming more visible. I’d already nailed my colours to the mast, by proposing that search, subscription, conversation and fulfilment were the “four pillars” of enterprise software around that time, so I was comfortable with what was happening. But I was still keen on understanding more about why, and wanted to do this in the context of transaction costs.
For some years, I had been playing with models for managing systems estate change; I was particularly keen on a principle I called “Spectrum”, where I could visualise the firm’s architecture as a series of loosely coupled layers, at one extreme touching the customer, and at the other touching the darkest denizens of back office operations. The idea was to colour-code clusters of systems using the visible spectrum, while declaring what happened on customer desktops “ultraviolet” and what happened at exchanges and payment mechanisms “infrared”. In between, “violet” represented apps that touched the customer, a layer exhibiting rapid change, and “red” represented accounting apps, a layer exhibiting glacial change. And everything in between to cover pre-trade, trading, post-trade, risk and settlement. The idea behind “Spectrum” was that an app could only be changed at the pace consistent with the layer it inhabited; it could ask for change in layers “above” it, layers that exhibited faster propensity to change, but had no right to request speedy changes to apps in layers “below” it; apps in each layer had to respect the rate of change associated with apps in slower layers. As a consequence, in my then utopian style, I had hoped to minimise the regression-testing logjam of enterprise architecture; we’d already avoided Spaghetti Junction by going for a bus architecture rather than point to point interfaces.
What I’d established in my own mind was a growing belief that the issue was to do with rates of change and costs of change. Vertical integration paid off when the rate of change was low. Networked small-pieces approaches paid off when the rate of change was high.
And then the time came to move on from the bank, and the challenges I faced were different. Telcos were very much about stacks rather than ecosystems; enterprise buses were rare; and open multisided platforms too outrageous to consider (though we did!). But the debate of integrated stack and SOA versus ecosystem and Web 2.0 continued to intrigue me. [Now I know that's an oversimplification, that SOA should really be about a set of design principles rather than explicit technical implementations and reference architecture, but what I saw was largely less of the former and more of the latter.]
Fast forward to a couple of years ago, and Clay Shirky. Clay (like John, Nick and Dion, someone I read regularly) wrote something about the collapse of complex companies, and referred to the work of Joseph Tainter in the process. While I’d heard of Tainter, I hadn’t read his work in depth, and I proceeded to dig into The Collapse of Complex Societies. [It was a subject I'd been mesmerised by since youth].
That led on to my finding other pieces by Tainter, including the diagrams below:
The first, above, looks at productivity of the US Healthcare system between 1930 and 1982. (They define productivity index as life expectancy divided by the ratio of health expenditure to GDP). [I must admit I was reminded of this chart when I came across the Hagel/Seely Brown/Davison Big Shift thinking a year or two ago.]
The second, which actually occurs earlier in Tainter’s paper, seeks to model diminishing returns to increasing complexity. Both diagrams are taken from Tainter’s Complexity, Problem Solving and Sustainable Societies, 1996.
And so to today. Development backlogs are endemic, as the sheer complexity of the grown-like-Topsy stack slows the process of change and makes it considerably more expensive to change. The stack has begun to fossilise, just at the time when businesses are hungrier for growth, when the need to deliver customer-facing, often customer-touching, applications is an imperative.
Which makes me wonder. What Tainter wrote about societies, what Shirky wrote about companies, are we about to witness something analogous in the systems world? A collapse of a monolith, consumed by its own growth and complexity? As against the simpler, fractal approach of ecosystems?
Just wond’ring. I will probably start taking a deeper look at this; if any of you knows of references worth looking into, please let me know.