Last week, I seem to have seen a statement from Alistair Cockburn (I’ll let you translate), one of the signatories of the agile manifesto, and one of the people who continues to feed the reflection, so a statement from Alistair Cockburn saying he had never seen a successful agile transformation in his career, implied at large scale, across the entire company.

Indeed it’s very difficult to judge, to succeed at, to measure. We know that only 30% of transformations (of all kinds) over the past 30 years succeed. But succeed means what? We can easily imagine, depending on the size of organizations, that it cannot be identical everywhere.

We’re not chasing a chimera otherwise we would have stopped long ago fighting. We observe the benefits of our approaches otherwise we would have stopped long ago too. The game is worth the candle. But nothing is simple, we knew that. So I asked myself the question again, pragmatically. In fact it was a client who asked me the question. The answer seemed so obvious to me that at the time I was left a bit frozen. So the answer I’m giving you here is not from an out-of-context reflection, but very concrete to a client, in a competitive environment, in a highly computerized setting.

What is a successful agile transformation?

More broadly, what does successful agile mean? In my eyes, it’s a capacity for companies to respond to the complex modern world: unexpected, intertwined, changing. Respond means having the best effort/value ratio. Value means things that generally sell, and therefore are either in line with market needs (not too late, not too early), or innovative (the first ones), or recognized for their qualities (adapted, durable, more effective, etc.). I remind you that I work 99% of the time in a competitive environment.

Agile, and one of its predecessors, Lean, thinks this will come from a capacity to regularly deliver finished things (feedback), a capacity to improve regularly. Agile also thinks that the performance lever for all this, where the difference will be made, is in the involvement and empowerment of people. Performance, a hated word, is more prosaically, being well and good at what you do.

An agile company is a company that knows how to respond this way (effort/value compromise -> market needs, innovation, qualities) by implementing regular feedback and continuous improvement and by relying on the empowerment and involvement of people.

This transformation can be equipped with several methods: scrum, kanban, xp, etc. These are recognized and validated tools that can help achieve these goals and these means (involvement, continuous improvement, etc.).

How to measure a successful agile transformation?

We need to know how to measure our target: innovative products and/or responding to needs and/or with qualities. We need to know how to measure our performance lever (being well and good): involved and empowered people, capacity to regularly deliver finished things, capacity to improve regularly.

Should we measure?

We need to know, to have feedback. What we call measurement is feedback and comparisons.

Measuring the target

If value is cash

The sales/cost ratio: it’s very difficult to have precise figures, there are many games with numbers and the total cost is often very difficult to make apparent. However we can have broad lines: by business domain, total time spent on maintenance / development (likewise the calculation of this load should be macro), etc. This must remain in broad strokes, because time measurements never work if they imagine themselves precise, and produce a disastrous effect when followed closely.

If value is product quality

These qualities will have an immediate effect on cash/sales. A regular customer survey on customer satisfaction would be a good means. A survey both on the product and on the relationship with the company. One of the product’s qualities is to have very few bugs: measuring client bugs is a good indicator (Number of bugs reported by clients (with no associated bonuses otherwise arbitrations on the term “bug” will be manipulated)).

If value is innovation

Measure the capacity to sell business domains, or user behaviors that don’t yet exist in the current offering (with no associated bonuses otherwise arbitrations on the term “new” will be manipulated).

Measuring the lever

Delivering finished things (that are used and probably sold): measure the time between the idea and delivery. Improving continuously: tracking improvement proposals and their completions.

Involved and empowered people: For this I suggest an anonymous regular questionnaire. This questionnaire is based on the proposals of Michael Sahota & Olaf Lewitz, it is necessarily anonymous, we’re going to (we’re just starting) have it filled out every two months, it’s the variations and direction that count.

Here are the questions (answers are based on a scale of 0 to 10).

• Trust level: I cannot trust (not at all = 0), I can trust (totally = 10)
• Security level: I feel insecure (completely = 0) or secure (completely = 10)
• Openness level: I can open up, pour out, (totally = 10) or I cannot (at all = 0)
• Connectivity level: I work with people (lots =10), or alone (all alone =0)
• Wholeness level: I flourish completely (10) or just on certain aspects of my being (not at all = 0)
• Power is distributed or centralized: from 0 (very centralized), to 10 (very distributed)
• Access to information: from 0 (very limited) to 10 (very transparent and easy to access)
• Planning, control or emergence: from 0 (very planned/controlled), to 10 (very emergent)

For this I based myself on the session with Michael Sahota & Olaf Lewitz that I attended, itself very inspired by Clare Graves' Spiral Dynamics. Here are the slides. Then you can use their table to place your organization and see it evolve over the months.

Should we measure the deployment of methods (Scrum, Kanban, XP)?

Yes and no. Yes because it will be difficult to measure anything else at the beginning. No because it’s not at all an indicator of the success of the company’s expectations, just an indicator, and not an objective or an ambition.

Should we measure “PMO” aspects?

That is: are projects going well and moving forward? Beware projects can “go well” and “move forward” to ultimately deliver nothing.

If a project regularly delivers things that are used (and therefore that we sell) and we have a very low client bug rate, and we measure the cash coming in, it’s useless to measure anything else. The main risk is believing we have something and we don’t have it: this risk is regularly mitigated by delivery to customers (and otherwise we get a fail fast). But it is useful to also know if a project is not going well to take action, make arbitrations. We must measure that projects do regularly deliver things (through to the end, and with qualities), and that they are not going badly. One can be induced by the other.

Indeed a score card (like at Spotify) can give a macro picture of the state of affairs of projects. We can create a sort of maturity level for agile teams.

Because the real PMO answer is not to measure projects but teams, and particularly their well-being and autonomy.