A complexity conscious approach to measure your Agile team’s productivity
A purely quantitative targets-based approach sets up teams to compete with each other instead of creating a culture of trust and collaboration. There might be a better way.
I worked for a large financial services client on a year-long assignment a few years ago. It was complex development work, with everyone focused on improving the predictability and productivity of the teams. Regulatory deadlines, tricky business flows, unpredictable team dynamics and a complex multi-vendor environment made it difficult for the leadership to have any meaningful, quantifiable metrics-based approach to steer the project.
However, that did not deter them from building a long list of metrics to track with numbers — especially the ones around team productivity.
As one of the leaders put it, “The number one thing is to make sure we are efficient; we have to do more for less. Make sure the teams are productive.”
Productivity and Predictability
In Agile environments, productivity is often a taboo word, and coaches often advocate choosing predictability over productivity.
The Cambridge dictionary defines productivity as the rate at which a person, company, or country does useful work and predictability as the state of knowing what something is like when something will happen.
Complex systems are inherently unpredictable. You cannot do an intervention X at one place and expect an outcome Y to happen at another. There are too many factors that impact those outcomes. The rearview mirror only reflects what went by, but you don’t know what lies ahead — especially when you are in turbulent weather.
[If you need a primer on complexity first, read these first
In software development, many think team velocity measured by hours or story points is a good measure of predictability or productivity.
On both these fronts, it’s a mirage.
Velocity-driven planning assumes that the amount of work in future and its complexity will be roughly the same as in past sprints. Also, you can’t say anything about the quality of those features by velocity alone. So any comment on the team’s predictability and productivity based on those numbers is a little problematic and challenging.
Perspectives define our definition of what’s good.
While velocity might seem helpful to managers, it might be a different story from the perspective of a product owner who works with that team.
The numbers look great with team A because team A does precisely as the product owner wants them to. However, team B engages the product owner in active dialogue and helps co-create the features. That often means changes until the end, but the features eventually have much less rework. This approach contrasts with team A, which does a professional dialogue with the product owner about locking the requirements well ahead of the sprint so they can freeze the sprint scope and meet the sprint commitments. Which team do you think the product owner thinks is more productive?
This is an important aspect: what is perceived as a good outcome for the manager may not be a good outcome for the product owner. What is the learning from this?
Should we now go on a wild goose chase to find another metric that makes the product owner happy?
The story behind the numbers
There is never an ideal measurement framework. Whatever target you set, teams will chase it and adapt their behaviour to achieve those targets better.
Whether you call these metrics outputs or outcomes, they are still only part of the narrative — the team’s journey to those outcomes, and all they did to achieve those outcomes. There are many stories with insights the quantitative data would never reflect.
In complex systems, it is next to impossible to attribute outcomes to specific activities or interventions that happened. Still, there is always a good story to tell when a team delivers a feature that made them proud. Stories have far more insights for agile teams because the outcome results from collective effort. Everyone contributed to that crazy feature deployed last weekend to production or surviving the downtime disaster during Christmas. Each team member contributed in their way with their unique strengths.
While the developers in a team can tell very concrete stories about their contribution to building the features; the story around their contribution to the team’s velocity is often theoretical. The qualitative discussion with stories goes together with a quantitative approach to outcomes.
Stories are the canvas for the picture the numbers portray, and they help you understand whether the picture is genuine or counterfeit.
Qualitative must coexist with quantitative.
In complex systems where you can’t predict, learning is crucial. It is at the heart of the probe-sense-respond approach advocated by the Cynefin framework.
Learning is also core to the agile mindset.
To make better sense, you need those stories, as they help you understand what’s going on. Stories assist you in understanding things better and are also a very humane approach to management. Teams feel there is an appreciation of their work beyond the numbers.
A balanced outcomes approach that relies on qualitative and quantitative measures is best suited for the complex systems most agile teams operate in often. You are likely to do more harm than good if you stick solely to number targets and attribute specific interventions to the success of those outcomes, and complex systems don’t work that way.
Five sprints to finish a story? Unbelievable !!
One of the teams I worked with was building a feature that commenced development in Sprint 8. Once the developer picked up the story, she ran into unforeseen problems. Some of the assumptions around the technical framework did not hold anymore. She would first have to do a spike to validate the new approach they designed. While the spike helped the team to uncover certain things and seemed helpful, the new approach had several performance challenges and was abandoned eventually.
Later, with help from some business users and the product owner, they simplified the requirements. The simplified requirements and more complex technical work saw the feature seeing the light of the day. This feature development turned out to be a long-drawn exercise that started in sprint 8 and only finished in Sprint 13. That story also hit the team’s overall velocity since they were all distracted and focused on getting this bugger out of the door.
Did the numbers paint an ugly picture in management reviews? They did.
But luckily, the key stakeholders did understand software development is a complex endeavour and were empathetic enough to hear the story behind the numbers in each review. This engaged interaction helped the team maintain confidence, improve, and achieve the outcomes. If they would just get pressured with the number game, the result would have been far different. After the slight dip, the team became the most productive in the program for many more sprints.
Did your perspective on this team change after listening to the narrative than just reading the subtitle. Do you relate better to it? Would you have thought this team was lousy and incapable without the story? Or just picked up something beyond their calibre?
It does not mean you are sympathetic to the team; go soft on them and let go of inefficiencies. You understand better to deal with them rather than just with the numbers. We also did that when we realised that the team needed additional expertise to overcome this bump and had to fix a “people issue” with one of the business analysts.
Understanding the story behind the numbers helps to make the interventions more concrete and relevant.
You cannot mandate productivity.
Productivity ultimately is an outcome, a result of the team doing many things. It can never be the starting point. As a leader, you must focus on what is in your control — ensuring that you create the best possible environment for the teams to thrive.
Steve Jobs puts it amazingly well when he says, “You cannot mandate productivity; you must provide the tools to let people become their best.”
Whether you use sprint velocity or sprint acceptance ratio or any other that makes sense for your context, a complexity-conscious approach has far more potential to measure and impact your agile team’s productivity in a meaningful way than a linear numbers-only approach.
It is an approach that helps you avoid setting up your teams to compete against each other in a number game, rather It lays the foundations to build a culture of continuous learning based on trust, collaboration and meaningful, measurable outcomes.
You might also want to read my book “Perspectives on Agility”, available on Amazon in print as well in Kindle format. Kindle Unlimited users read it for free. You might enjoy a quick summary of the book before you buy.