When To Stop Looking at Code Coverage – Goodhart’s Law

All companies want to know if their investment will bring them value.  In order to understand how much value they are getting, the company will implement some sort of measurement.  This is common sense: I invest, show me what I’m getting with my investment.  However, a measurement can have an adverse side affect in the creative process, particularly if it is a requirement.

Code Coverage

A number of years ago, Test Driven Development (TDD) spread throughout the software industry as a way to increase software quality.  Tests were written first, then the software was written to satisfy the tests.  To see how much of the software was covered by the tests, the test runners would give a number that would represent the percentage of how much of the code was covered by tests.

Research showed that this percentage correlated with high quality code in the industry. Because of that correlation, many businesses believe that they have to have a high percentage of code coverage, in order to get high quality code.  High code coverage was then set as a requirement for many software shops.

Goodhart’s Law

Economist Charles Goodhart, while criticizing the United Kingdom government of Margaret Thatcher in a 1975 article for trying to conduct its monetary policy on the basis of targets, created a simple law:

When a measure becomes a target, it ceases to be a good measure.

Goodhart saw that when a feature of the economy is picked as an indicator of the performance, then it inexorably ceases to function as that indicator because people start to game it. (https://coffeeandjunk.com/goodharts-campbells-law/)

Examples of Goodhart’s Law

This simple law is applicable to many industries.  Here are some examples of setting measurements as targets:

Search Engine Optimization – Once web masters found out that the quantity of keywords on their website would increase their search engine rankings, they would duplicate keywords countless times and have the text color be the same as the background color so that the users wouldn’t be confused about all the random text.

No Child Left Behind – Once schools understood that in order to get proper funding their students had to do well on standardized tests, the schools started teaching to the tests instead of continuing to learn content.

Dieselgate – In September 2015, the EPA found that Volkswagen had intentionally programmed turbocharged direct injection (TDI) diesel engines to activate their emissions controls only during laboratory emissions testing which caused the vehicles’ NOx output to meet US standards during regulatory testing, but emit up to 40 times more NOx in real-world driving. (Wikipedia)

When Code Coverage Is A Bad Indicator

Once the goal of a developer is set to hit a high code coverage number, the developer will get pretty inventive.

One shop I was at, that had code coverage as a requirement, I was spotting this all over in the codebase:

[System.Diagnostics.DebuggerHidden]

I tried to step debug into some code, but couldn’t.  The previous developers started to hide code from the debugger, in order to increase the code coverage percentage, since the test runner used the debugging framework to assess if the code was hit during testing.

This was obviously against the spirit of code coverage, but they were also under really tight deadlines. Having the measurement as a requirement made them do bad things to the system.

Overview

Once a measurement becomes a requirement for your dev team, it ceases to be a good measurement. Developers will start coding to the measurement instead of what’s really important: business value.