Why it’s Difficult to Measure Developer Productivity
Software engineering is creative and there are outside forces that influence productivity.
This week I read A Human-Centered Approach to Developer Productivity, the first article in a new series on developer productivity by Ciera Jaspan and Collin Green. Here, the authors explore the reasons why it is difficult to measure developer productivity, and set the stage for a more human-centered approach.
My summary of the article
Engineering leaders frequently want to understand whether productivity is up, down, or stable. They want “up” and “down” to map to “good” and “bad.” Unfortunately the task is not so simple.
Why is it so difficult to measure developer productivity? The authors suggest two reasons:
Engineers are humans; this is not a robot-powered factory
One problem that makes the measurement challenge difficult is that there are many outside forces that influence how productive a human will be at a task. An outside force could be the complexity of the work (and whether it is necessarily that complex), the interactions with others to get the job done, or the organizational design. There are also factors that specifically affect developers, including flaky tests, build speeds, and technical debt.
Software engineering is not predictable
Software engineering is a creative endeavor. It is not about the production of uniform, interchangeable outputs. Therefore:
Throughput cannot be conflated with productivity.
We cannot assume the product that was built is the right product. It may have performance issues or it may not scale; there are quality, reliability, and practicality aspects of creating software.
We cannot assume that when work doesn’t result in output, it has no value. Problem solving is a core part of software engineering and it is hard to measure.
Attempts to quantify work productivity by borrowing methods from operating machinery are not suited to software engineering.
Measuring productivity is complex, however there are still important reasons for trying. The authors recommend leaders measure productivity “in a holistic and multifaceted way, not in a reductionist, unidimensional way.” Select more than one metric, consider short- and long-term productivity, and remind stakeholders that they are measuring a fundamentally human system.
Previously we learned about Google’s Goals/Signals/Metrics framework for selecting metrics. This article may serve as a helpful reminder for those approaching the last step, choosing metrics: consider the different aspects affecting developers’ jobs and collect a holistic set of metrics.
I’ll conclude with a quote related to this from Rebecca Murphey (ex-Stripe, Developer Productivity):
“In reality, this isn’t a robot-powered factory we’re dealing with: we’re trying to quantify the value of the output of an organic, emergent, and fundamentally human system, one that is often affected by “outside” forces considered generally interruptive and immutable. In the course of trying to define developer productivity such that you can boil everything down to one number, you may look back and realize you’ve done very little to make anyone more productive.” — Rebecca Murphey, Measuring developer productivity
That’s it for this week! Share your thoughts and feedback anytime by replying to this email.
If this email was forwarded to you, subscribe here so you don’t miss any future posts:
That article resonates with me. It is also really similar to what others call “SPACE” metrics https://queue.acm.org/detail.cfm?id=3454124