Code Coverage Is a Vanity Metric and You Know It

A team at 95% coverage with brittle tests has worse quality than a team at 60% that tests what matters. The dashboard is lying to you in green.

The hook

Two weeks after my team hit their quarterly OKR of 95% code coverage, a payments bug made it to production. A Stripe webhook handler was silently dropping events when the payload contained a nested subscription with more than one item. The handler had a test. The test used a snapshot: a JSON fixture captured six months ago from a single-item subscription. The test passed. The code was covered. The bug shipped. I stared at the Codecov dashboard, glowing green, and realized we had spent a quarter optimizing for a number that meant nothing.

अवगुणांचे हाती · In the Hands of Bad Qualities

अवगुणांचे हाती । आहे अवघी फजिती ॥
नाही पात्रासवे चाड । प्रमाण ते फिके गोड ॥
तुका म्हणे भाव । शुद्ध बरा सोंग वाव ॥
When the inner substance is flawed, outward show amplifies the embarrassment. Pure intent beats performance.

What I keep seeing

I pulled up the Codecov report and filtered for the payments directory. Coverage: 97%. I opened the test files. Of forty-seven tests, thirty-one were snapshot tests generated by a Copilot-assisted workflow the team adopted to "boost coverage fast." Each snapshot asserted that output matched a frozen fixture. None tested behavior at boundaries. None sent a malformed webhook. None verified that idempotency key logic actually prevented duplicate charges.

This is not a story about one team. Istanbul, c8, Codecov, Coveralls: the entire coverage toolchain measures which lines were executed during a test run and presents it as a proxy for whether the software works. The proxy became the target. Goodhart's Law, doing exactly what it always does.

The mechanics

Code coverage tools instrument your source by inserting counters at branch points and statement boundaries. When a test runs, the counter increments. After the suite finishes, the tool divides executed statements by total statements and produces a percentage. This percentage tells you which lines were touched. It tells you nothing about what was asserted.

A test that calls a function and never checks the return value registers as coverage. A snapshot test that freezes the current output, correct or not, registers as coverage. A test that mocks every dependency and only verifies the function does not throw registers as coverage. In each case, the metric improves. In none of them does the test catch a regression.

Mutation testing (Stryker for JavaScript, mutmut for Python, pitest for Java) addresses this by modifying your source and checking whether your tests fail. Flip a conditional from > to >=. If no test fails, you have a surviving mutant: proof that your tests do not guard that behavior. The difference between coverage and mutation score is the difference between "the doctor looked at you" and "the doctor diagnosed you."

Where Tuka comes in

Tukaram wastes nothing here. Three lines. "अवगुणांचे हाती, आहे अवघी फजिती" (in the hands of bad qualities, all is embarrassment). The structure is causal: when the substance is wrong, the display does not compensate. It makes the failure louder. A team proudly posting 95% coverage in Slack when the tests behind that number are hollow is performing exactly the dynamic Tukaram describes. The higher the number, the more visible the eventual collapse.

His closing line is the one worth screenshotting: "भाव शुद्ध बरा सोंग वाव" (pure intent is better; pretense is wind). Bhav in a test suite means: does this test express a belief about how the system should behave? If yes, it has substance. If it merely records what the system currently does, it is pretense. It fills the metric. It holds nothing.

What I would actually do

Replace your coverage gate with a mutation score gate. Set up Stryker in CI, target it at critical paths: payments, auth, data pipelines. Threshold: 70% mutation score, not 95% line coverage. Run mutation testing nightly or on merge to main, not on every push. Remove coverage percentage from your team dashboards entirely. Put mutation score and "tests that caught a real bug this quarter" in its place. When an engineer writes a test, the question is not "does this raise the coverage number." The question is "if this code broke tomorrow, would this test tell me." If you cannot answer that in one sentence, the test is wind.

Chetan Dhandal