Testing What Matters, Not What's Easy

We treat the coverage badge like a quality bar. We raise the gate when leadership asks for "more testing." We celebrate when the PR check goes green at 80, then 85, then 90. We write another test for a DTO mapper because the uncovered lines are easy and the refund reconcile is annoying.

Then production fails on a path the suite never really exercised — and the dashboard still looks fine.

That's the coverage trap: a green percentage that measures execution, not protection.

The Coverage Trap — Green Bar, Dead Path

The badge that passed

The report looked honest. Checkout service: 82% lines, branch coverage "acceptable," CI gate satisfied. Dozens of unit tests around the Stripe webhook handler. Status codes. Payload shape. Happy-path event types. Mock client. Sub-second suite. Reviewers stopped scrolling once the badge went green.

Nobody asked which production failure the suite would refuse to ship.

The path that never ran

Stripe retried a payment_intent.succeeded event. The handler marked the order paid again — or fired a second side effect — because the idempotency key check lived in a branch the mocks never forced.

Lines around the check were "covered." A test called the function with a fresh key and asserted 200. The expensive branch — same key, second arrival — stayed theatrical. Covered in the numerator. Dead in production.

I've seen the twin on refunds: reconcile job "tested" with a single successful ledger write, while the double-credit path on a partial Stripe refund never got an assertion. Same badge. Different invoice.

Auth has its own version. Session refresh covered by a test that always injects a valid token. The expired-token + concurrent-refresh race never runs. Coverage still climbs because the happy function body is long and easy to hit.

Three paths. One metric. Only the cheap ones feel "done."

Why the numerator lied

Coverage credits a line when a test executes it. It does not ask whether an assertion would notice if that line started lying.

Google's coverage practitioners put it plainly: coverage guarantees execution, not that lines or branches were tested correctly. Production can still break because you missed a path or because you covered an area and still skipped the edge case inside it.

Partial tests can still walk every branch and leave the failure mode that costs money unguarded. That is why mutation tooling treats line coverage as a weaker signal — a suite can execute everything and still fail to detect seeded faults.

High coverage can make the critical path less visible. Getters, config loaders, and mapper glue are cheap hits. They inflate the same percentage that should have been warning you about refund reconcile and auth edge cases.

The bar goes up. Risk concentrates in the unglamorous branches.

The Gate Invents the Wrong Work

Make a number a target and people will hit the number. Same failure mode as turning DORA into scoreboard KPIs.

An 80% gate does not say "protect checkout." It says "make the uncovered set smaller." Engineers are rational. They clear private helpers and serialization edges before they build a slow, environment-heavy test for webhook idempotency.

Fowler's absurd case is assertion-free testing — execute the code, assert nothing, watch coverage rise. The everyday case is milder and more common: tests that look for things that rarely go wrong and distract from the things that do.

Google's writers land the same punch from the other direction. Chasing the number toward 100% creates false security and a pile of low-value tests you now have to maintain. Pressure to hit the metric almost never yields the outcome you wanted. Treat a gate like a checkbox and it backfires.

The queue shape changes under the gate. Ticket titles become "cover Utils.formatMoney" instead of "prove duplicate webhook is a no-op." Standups celebrate green PRs. On-call still owns the money path.

The gate invents work that is easy to count.

Path risk invents work that is easy to ship wrong.

Risk Map Over Percentage

Once you already have a non-trivial suite, another five points of coverage is usually the wrong next investment. Replace "what's our %" with three cheaper questions.

Failure-cost tiers

Map paths by what a failure costs, not by how many lines they contain.

Take a thin checkout slice:

• Cheap fail — catalog browse returns a stale cache entry; the user refreshes.
• Expensive fail — payment capture doubles a charge or drops paid status.
• Rare expensive — refund reconcile mismatches the ledger; finance finds it next week.

Coverage equalizes them in one percentage. A risk map does not. Put engineering time on expensive and rare-expensive first, even if that leaves a getter red on the report.

If you need a ritual, do this in planning: list five paths that would page someone or move money wrong. Assign each a tier. Only then open the coverage HTML. Untested cheap paths can wait. Untested expensive paths cannot hide behind an 82% headline.

Critical-path exercised?

Binary beats percentage for money and auth.

For each critical path, ask one question: did a test run the real branch with the failure shape production hits — duplicate event, expired token, partial outage — or only the happy mock?

If the answer is "mock only," that path is not exercised. Call it uncovered in the risk sense, whatever Istanbul prints. The webhook example above was 82% covered and zero for duplicate delivery. That is the whole critique in one service — cousin to letting a coding agent double as tester.

Write the answers down next to the paths. Yes / no. No averages. Averages are how the trap returns wearing a spreadsheet.

When mutation survivors matter

Mutation testing seeds small faults and asks whether any test fails. Survivors are concrete blind spots: change >= to >, delete a guard, force return true — suite still green.

Stryker's age-gate demo is the same shape as the webhook story. Mutate the guard to always allow entry; if tests still pass, you never asserted the reject case. PIT frames the same idea as the gold standard against which other coverage types get measured: execution is not detection.

You do not need a monorepo mutation score as a vanity metric. Run it on billing, auth, and the packages that page on-call. Read the survivor list before you worship the aggregate. Google points at mutation research specifically when you need to know whether covered lines were asserted, not merely executed.

Keep Coverage. Demote It.

Delete the gate only if your team already treats testing as a habit. Low coverage still means large areas never run under automation — that signal is real. Fowler treats numbers below roughly half as trouble. Google treats low coverage as a guarantee that big regions ship untested on every deploy. Keep that alarm.

Demote the number from quality bar to smoke signal. Useful when you are climbing out of almost-nothing. Increasingly noisy as a pride metric in the upper 80s.

Google's informal bands — 60 acceptable, 75 commendable, 90 exemplary — are guidelines, not a universal law. They also warn that obsessing from 90 to 95 is usually the wrong fight next to climbing out of 30 — same patience as an incremental strict-mode path. Prefer human judgment over the uncovered lines in review. Prefer new-code or delta gates over a frozen org-wide checkbox if you must gate at all.

Coverage finds gaps. It does not certify the paths that matter.

If a suite is weak in a way coverage can see, it is probably weak in a way coverage cannot.

Ship the risk map and let the badge follow.