Skip to content

Lesson 7 — Delivery Metrics and Scaling

At a glance

Learning objectives

By the end of this lesson you should be able to:

  1. Distinguish measures that inform decisions from measures that flatter, and explain why a measure used to judge people stops being a measure.
  2. Define a small set of delivery and quality measures precisely enough that two people would compute them the same way.
  3. Explain what flow measures (work in progress, cycle time, throughput, work item age) reveal that cost and schedule measures do not.
  4. Reason about when scaling a delivery model helps and when it only adds coordination cost.
  5. Describe what changes at scale — planning cadence, interfaces, architecture alignment — and why architecture and team structure constrain each other.
  6. Recognise the failure modes of scaling, and state the limits of the scaling literature honestly.

Prerequisites

  • Module 11 Lesson 1 — measures, baselines, and the difference between leading and lagging indicators.
  • Module 04 Lesson 3 — work-in-progress limits and flow, which this lesson extends from a team to several.

Introduction

Two related questions occupy experienced delivery managers, and both are places where confident advice outruns evidence. The first: what should we measure? The second: should we scale this way of working beyond one team?

They belong together because the same trap catches both. In measurement, the trap is optimising what is easy to count. In scaling, it is copying the practices of a larger organisation without the conditions that made them work. Both produce activity that looks like improvement and is not.

1. Measures that inform, and measures that flatter

A measure informs when it can change a decision. Everything else is decoration, and some decoration is actively harmful.

Type Example What it does
Informing Work item age, change failure rate, cycle time trend, defect escape Reveals where work is stuck or where quality is escaping
Flattering Story points delivered per person, lines of code, hours logged, number of meetings Rises when nothing improves, and falls when people work sensibly
Distorting Any measure used to compare individuals or teams for reward It becomes a target, and the measure stops measuring the thing

The third row deserves its own name. Goodhart's law is the general statement — a measure that becomes a target ceases to be a good measure — and the operational version is simpler: if you attach individual consequences to a measure, people will optimise the measure. Velocity compared between teams produces point inflation; defect counts used punitively produce fewer reported defects; utilisation targets produce people who look busy and finish less work.

Three rules make a measure set usable:

  1. Written definition. "Cycle time" must say from which event to which event, and which items count. Two teams measuring undefined cycle time cannot be compared even informally.
  2. Used for learning, not ranking. Report measures for a team's own trend, and use the same definition across teams only to find systemic problems, never to rank.
  3. Retired when it stops informing. A measure nobody has acted on in three months is a reporting cost.

2. Four families of delivery measures

Family Example measures What it is for
Flow Work in progress, cycle time, throughput, work item age Finding where work waits and how long delivery takes
Delivery stability Change lead time, deployment frequency, change failure rate, time to restore service Whether delivery is fast and safe
Quality Defect escape rate, reopen rate, defect age, cost of rework Whether verification is working, and what poor quality costs
Predictability Variance between forecast and delivered work, milestone slip count Whether commitments can be relied on

The four measures in the second row are the ones the DORA research programme popularised and that Accelerate (Forsgren, Humble & Kim, 2018) reports on: change lead time, deployment frequency, change failure rate, and time to restore service. Their value in this course is conceptual rather than comparative: they measure speed and stability together, which makes visible the common trade of delivering faster by breaking more things. If you measure only throughput, that trade is invisible.

Two cautions about applying them. They assume an automated delivery pipeline and frequent deployment; where a release is a quarterly event, deployment frequency and lead time carry much less information, and a measure that cannot move is not worth reporting. And they are team or service measures, not individual ones — using them per developer produces exactly the distortion described above.

Flow measures complement them and are cheaper in every environment. Work item age — how long the oldest in-progress item has been open — is arguably the most useful single weekly number a team can look at, because it exposes the work that has been forgotten rather than the work that is visible.

3. What changes when delivery scales

"Scaling agile" is usually discussed as a choice of framework. The mechanics matter more, and they are architecture and coordination questions before they are process questions.

Dimension What changes as teams multiply
Interfaces Any change spanning two teams needs an agreed, versioned interface — in the same way a supply chain does
Planning cadence A shared rhythm is needed for cross-team dependencies; without it, teams plan against versions of each other that do not exist
Architecture alignment Team boundaries and system boundaries interact (Conway's law, 1968): the interface structure of the system tends to mirror the communication structure of the organisation
Integration More teams means more integration points, and integration cost grows faster than headcount
Coordination cost Every additional team consumes attention from the others; the added communication can exceed the added capacity
Decision-making Central control cannot respond fast enough, so authority must be delegated — with the decision rights table from Lesson 1

The framework question sits on top of those mechanics. There are several published scaling frameworks — SAFe, LeSS, and Scrum@Scale among them — and they make different assumptions: SAFe prescribes a substantial organisational structure and cadence, LeSS keeps a single product backlog and adds structure minimally, Scrum@Scale leaves the shape to the organisation. Choosing between them is a context-dependent decision: how many teams, how coupled the product is, how much regulation the organisation carries, and how much change the organisation will actually absorb. What none of them can do is supply capability the teams do not have.

The honest statement about this literature is that it is largely practitioner-authored rather than independently researched. Accelerate is one of the few sources in this space based on survey research rather than practice reports, and its central finding — that delivery performance is associated with technical practices and organisational culture rather than with any particular framework — is a useful counterweight to framework advocacy from any direction.

4. When scaling helps, and when it does not

Situation Scaling helps Scaling harms
One team, stable requirements, clear priorities No — nothing to coordinate Adds cadence and artefacts to a team that was fine
Several teams on one product with real dependencies Yes — interfaces, shared cadence, cross-team planning It does not help if the teams do not share a product or an integration
Many teams across an organisation, different products Valuable at the portfolio layer: priorities and capacity, as in Lesson 2 A single scaled process imposed on unrelated products
A team that cannot deliver predictably No — scaling multiplies the problem Certain: several unpredictable teams are worse than one
Highly coupled architecture Only with architectural work to reduce coupling Coordination replaces integration testing, and the release still stalls

Two failure modes are common enough to name:

  • Scaling before capability. Adding structure to an unpredictably delivering organisation multiplies the coordination cost without fixing the delivery problem. The usual prescription is the opposite: fix reliability in one team first, then replicate what works.
  • Synchronised theatre. A quarterly planning event for hundreds of people, at which dependencies are identified in a two-day workshop and never resolved. Big-room planning is not useless — it can surface interfaces — but its value depends on whether the dependencies it surfaces are subsequently owned.

And one that is easy to miss: scaling that adds more coordination than it removes. Every new meeting, board, role, or report consumes capacity. If the coordination added exceeds the coordination it replaces — the informal contacts it destroys, the decisions it centralises — the organisation has scaled its overhead rather than its delivery.

5. What none of this does

No measurement system, and no scaling framework, substitutes for:

  • Knowing whether the product is worth building. Delivery measures say nothing about value.
  • Capable people in sufficient numbers. Staffing is a constraint, not a metric.
  • Architecture that supports change. Where the system cannot be changed safely, delivery measures describe the problem rather than solving it.
  • A decision-making environment that can decide. Slow governance caps delivery performance regardless of how work is organised.

Key terms

Term Meaning
Informing measure A measure that can change a decision
Goodhart's law When a measure becomes a target, it ceases to be a good measure
Flow measure A measure of how work moves: work in progress, cycle time, throughput, work item age
Change lead time Time from a change being started to it running in production
Change failure rate The proportion of changes that cause a degradation requiring remediation
Defect escape rate Defects found after the stage at which they should have been caught
Work item age How long an individual item has been in progress
Conway's law The observation that system structure tends to mirror organisational communication structure
Coordination cost The capacity consumed by communication and alignment between teams
Scaling framework A published model for coordinating multiple teams on one product

Worked example

Task: decide what to measure, and whether to scale, for a bank's digital division running three teams on one mobile banking application. Figures are invented for teaching.

Step 1 — Start from the decisions, not the data. The division has three decisions it makes badly today: whether to release, whether to add capacity, and whether quality is improving. Each decision needs at most two or three measures, and anything else is reporting cost. The measure set is therefore: for release, change failure rate and time to restore service; for capacity, flow — work in progress and cycle time per team; for quality, defect escape rate and reopen rate. Six measures, each with a written definition.

Step 2 — Write the definitions before the first collection. "Cycle time" is defined as from the item entering the in-progress column to reaching done under the agreed definition of done. "Change failure rate" is defined as the proportion of production changes requiring a remediation that is not routine configuration. The definitions are written because three teams reporting "cycle time" with three different starting points would produce a comparison that means nothing — and the temptation to compare is exactly what the definitions prevent.

Step 3 — Establish a trend, not a benchmark, and refuse to rank teams. Each team sees its own twelve weeks of data. The three teams' numbers are brought together only to identify shared constraints: all three report a rise in work item age for changes touching the shared authentication component. That is a systemic finding, and it points at an architectural constraint rather than at any team's performance.

Step 4 — Quantify the coordination cost before scaling. The three teams are on one product with real dependencies — 9 of 22 changes in the last quarter touched the shared component. A shared planning cadence of one hour a week costs roughly 0.025 full-time equivalents per team including preparation. The proposed alternative — a two-day quarterly planning event — costs about 0.05 FTE per team per quarter plus preparation, so it is cheaper in capacity and much less useful for dependencies that arise between quarters. The comparison of coordination cost against dependencies resolved is what makes scaling a reasoned choice rather than a fashion.

Step 5 — Choose the minimum structure that addresses the actual constraint. The constraint is the shared interface, so the response is: one owner for the shared authentication component, a versioned interface with a published change route, and a weekly 30-minute cross-team dependency review. No framework is adopted, no new roles are created, and no portfolio board is introduced — because the problem is one interface and the governance already exists (from Lesson 1). A scaling framework would have been a defensible choice in a larger division; here it would add cadence to three teams that already coordinate.

Step 6 — Set the review conditions and the honesty note. The arrangement is reviewed after eight weeks against three questions: did work item age for shared-component changes fall, did the dependency review resolve dependencies rather than list them, and did any team's cycle time worsen? The written note records that three of the six measures are new, so no baseline exists and the first eight weeks establish one — rather than inventing a target that the data cannot support.

Common misconceptions

“We should measure productivity”

Individual productivity measures in software generally measure activity: points, lines, hours, tickets. They rise when work is split into smaller pieces and fall when people take time to think, so they point in the wrong direction as often as the right one. Flow measures — cycle time, work item age — describe the system, which is what a manager can actually change.

“Our framework will fix delivery”

Frameworks supply coordination structures and vocabulary; they do not supply capability, architecture, or decisions. Where a team cannot deliver predictably, adopting a scaling framework typically adds meetings to the problem. Fix reliability in one place, then replicate what worked.

“If we measure it, it will improve”

Some things improve with visibility — long-waiting items, defect escape. Others distort the moment they are watched: velocity per person, defect counts under punitive use, utilisation. The question is not whether to measure, but what the measure will be used for.

“More teams means more delivery”

Beyond a point, added teams consume more coordination than they contribute, particularly on a coupled product, because integration cost grows faster than headcount. The alternative to adding teams is reducing coupling: better interfaces and clearer ownership, which is architectural work rather than organisational restructuring.

Practical activity

Choose measures for a project you know, and test its scaling assumptions.

  1. List the decisions your project or team makes weekly. For each, name at most two measures that would change it — and delete the rest.
  2. Write precise definitions for three of them, including the start event, the end event, and what counts.
  3. Identify one measure currently collected that is either flattering or distorting, and say what you would replace it with.
  4. If more than one team is involved, count the cross-team dependencies in the last period and estimate the coordination cost of the current arrangements in person-hours. Compare them.
  5. Choose the minimum structure that addresses the actual constraint — an owner, an interface, a cadence — and state the conditions under which you would revisit it.

What "good" looks like: every measure tied to a named decision, three definitions precise enough for two people to compute identically, at least one measure removed, a coordination cost quantified rather than asserted, and a structure chosen because of a constraint you can point at.

Self-assessment

  1. What makes a measure "informing", and why do cost and schedule variance often fail that test during delivery?
  2. Why does using a measure to compare teams change the measure?
  3. What do flow measures reveal that defect counts and budget variance do not?
  4. Give three conditions under which scaling a delivery model makes things worse.

  5. Why does Conway's law matter when deciding how to organise several teams?

  6. Apply it — metrics. A delivery director introduces a per-developer measure of story points delivered, publishes a leaderboard weekly, and reports that velocity has risen 30%. Cycle time has risen, defect escapes have doubled, and two developers have asked to be moved to another team.

    Hint: A strong answer explains what has happened to the measure and names a replacement set.

Show answers
  1. A measure informs when acting on it changes a decision — where to intervene, whether to release, whether to add capacity. Cost and schedule variance often fail that test during delivery because they report work already done: by the time they move, the response options have narrowed, so they describe the situation rather than changing it.
  2. Because people respond to consequences. If a measure affects how teams are judged, resourced, or rewarded, effort shifts toward whatever moves the number — splitting work into smaller items to raise velocity, or reporting fewer defects — so the number stops reflecting the underlying reality. Cross-team comparison also invites point inflation, which then destroys the measure's usefulness even within a single team.
  3. They show where work waits and how long individual items take. A rising defect count and a balanced budget can both coexist with an item that has been in progress for six weeks, or with a flow of work that is blocked at one step. Flow measures locate the constraint, which is what a corrective action needs to address.
  4. Any three of: teams that do not deliver predictably, whose problems are multiplied rather than resolved; products that are not actually coupled, so the coordination structures have nothing to coordinate; architectures whose coupling forces coordination that structure cannot supply; and organisations where the added coordination exceeds the coordination it replaces, so capacity is consumed rather than increased.
  5. Because system structure tends to mirror communication structure: if two teams own one interface, the interface's quality and stability depend on how those two teams communicate, and reorganising teams without changing the system's interfaces usually relocates the problem rather than solving it. It is a strong reason to make interface ownership explicit whenever team boundaries change.
  6. The measure has stopped measuring what it describes: attached to individuals and used to rank them, story points become a target, and the observable effects — inflated estimates, items split to count more, quality deferred — are exactly the rising cycle time and defect escapes in the report. Velocity is also a relative, team-specific measure, so comparing it across developers compares nothing. Replace it with a small defined set used for learning rather than ranking, spanning the four families: flow (work in progress, cycle time, throughput), delivery (released increments, lead time), quality (defect escapes, rework share), and outcome (whether the increment moved the measure it was meant to move). Write the definitions precisely enough that two people compute them the same way, keep the measures at team level, and retire any measure that no longer changes a decision. The two departures are also information: a leaderboard that makes people leave is reporting on the organisation, not on them.

Summary

  • A measure earns its place by changing a decision; measures attached to individual consequences stop measuring what they describe (Goodhart's law).
  • Write definitions, use measures for learning rather than ranking, and retire measures nobody acts on.
  • Measure across four families: flow (work in progress, cycle time, work item age, throughput), delivery stability (lead time, deployment frequency, change failure rate, time to restore service), quality (defect escape, reopens, rework cost), and predictability (forecast variance).
  • Scaling changes interfaces, cadence, integration cost, decision-making, and architecture alignment before it changes process. Conway's law explains why team structure and system structure constrain each other.
  • Scale when there are genuine cross-team dependencies, not to fix unpredictable delivery. Quantify the coordination cost, and prefer the minimum structure that addresses the actual constraint.
  • The scaling literature is largely practitioner-authored. Delivery performance is associated with technical practice and culture rather than with any single framework, which is a reason to choose minimally and revisit on evidence.

Further learning

Verifiable sources for the material cited: Nicole Forsgren, Jez Humble, and Gene Kim, Accelerate (2018), for the research programme behind the four delivery measures; Melvin E. Conway's 1968 paper for the observation about system and organisational structure; and the published documentation of SAFe, LeSS, and Scrum@Scale, each maintained by its own organisation, for their differing assumptions — read as competing practitioner positions rather than as independent evidence.