Noise: The Other Side Of Human Error

Two felons who should be sentenced to six years for identical crimes are jailed for four and eight years, respectively. An insurance company overprices some policies and underprices others. One HR executive enthusiastically recommends hiring a candidate while another turns her down. 

These are examples of errors. But they are not the sort of errors we have come to associate with bias, prejudice, or partiality. Instead, these errors arise from mere variability. There are variations in judgments that should ideally be identical, but there is no obvious reason for the variability. This variability in judgments is what we call noise. 

Noise is a pervasive source of errors. Wherever human judgment exists, you’ll find it. Unlike bias, however, it is largely invisible. Like a stealthy predator, noise goes largely unnoticed, while wreaking havoc in our decision-making in all fields. 

Differences in judgment are of course not surprising by themselves. Indeed, whenever we say that something is “a matter of judgment,” we imply that some differences of appreciation are acceptable, even expected. We don’t say the sun will rise tomorrow as a matter of judgment; but we say that a coach makes a professional judgment when he drafts an athlete for the team. What is surprising – and problematic – is not the existence of differences in judgment: it is their magnitude. 

Intolerable Levels of Noise 

Take for example, insurance underwriters. Underwriters make judgments when they assess risks and price policies accordingly. At one insurance company, the executives in charge thought that the median difference between two experts looking at the same risk would be around 10%. This would be an acceptable difference, they said; and certainly, one that they could live with. But when we submitted the same cases to multiple underwriters – something the company, of course, does not routinely do -, we found the figure to be 55%! In this case, noise was five times greater than people expected and 550% more than the organization deemed tolerable. 

This is a typical finding. Time and again, the results of such experiments (which we call “noise audits”) reveal surprising, and often intolerable, levels of noise. For five years, professor of psychology Daniel Kahneman, legal scholar Cass Sunstein and myself have attempted to document examples of noise, to unravel its mechanics and to explore how it is seeping into our “neutral” forms of leadership.  

Noise Is Not Bias – But It Is Bad, Too 

Let’s take a step back and consider the judgments that different people make about a given professional problem – how to price an insurance policy, whether to hire a candidate, how much to pay for an acquisition, how long a project will take, and so on. Different people make different judgments. This means they also make different errors. Part of the errors are shared: that’s bias. For instance, we may suspect that people estimating the duration of a project will, on average, display an optimistic bias. But there is also a component of their errors that is not shared. That variability is noise. 

Noise impacts professional judgments wherever we expect consistency. Judges, doctors, teachers, insurance underwriters, or HR professionals do not just express their personal opinions: they are operating on behalf of organizations and make decisions that commit them. We assume, at least implicitly, that another doctor would give the same diagnosis, that another judge would mete out approximately the same punishment, and that another insurance executive would have offered roughly the same price. Evidence of noise shows this assumption to be misguided. 

This should be a source of concern for these organizations. It is a source of error: the insurance company will lose business when it overprices policies and lose money when it underprices them. It is also a source of injustice. If the decisions of the judicial system (or those of your company when it makes hiring decisions) are largely determined by the lottery that picks the person in charge of a particular case, this is not anybody’s idea of a just and fair system. 

Of course, errors can also be caused by bias. Indeed, bias and noise often coexist. A company, for instance, could have a gender bias in its hiring decisions. But, at the same time, it could also suffer from noise, if different recruiters make different decisions about the same women or about the same men. Both bias and noise are sources of error. 

Unlike bias, however, noise doesn’t have an obvious cause. The random scatter of judgments is harder to comprehend than a shared, systematic deviation. That is why, while bias is a well-known problem, noise usually remains unseen. Bias can be the star of the show. Noise is a bit player, usually offstage. 

Dealing With Noise 

In “Noise: a Flaw in Human Judgment”, we try to shed light on the problem of noise and how to deal with it. We provide several shocking examples of error and unfairness in fields as diverse as college admissions, hiring and firing decisions, child custody, criminal justice, fingerprint analysis and economic forecasting. However, our point is not just to blow the whistle on noise. We also want to equip readers with tools to deal with this important problem. 

The first step for an organization is to run a ‘noise audit’ to measure variability in professional judgments. We give many examples (and a detailed method) to help leaders conduct audits of their decisions. 

Once an organization has determined the extent of the problem, it will want to change its approach to decision making to reduce noise. This is what we call “decision hygiene”. The intentionally off-putting term means to convey a message: just like we cannot see every germ on our hands, we cannot identify every source of noise, but it doesn’t mean the germ doesn’t exist. The fight against noise, like the fight against germs, can only be fought through broad-spectrum prevention. Like washing your hands, noise reduction will prevent serious problems, although you will never know exactly which ones. 

“Noise” is published at a moment where the COVID-19 pandemic has shaken the world’s convictions to its foundations, thus exacerbating this feeling of being subjected to a lottery. In many ways, this health crisis brings noise into a harsher light. For example, European countries reacted differently to the same set of facts on the negative side-effects of one of the vaccines. Despite a shared objective – saving lives -, health authorities in each country took different decisions. We don’t know who is right and who is wrong, but if their conclusions are different, not all of them can be right. This is an example of noise in a domain where we would usually not be able to see it. By using more of the decision hygiene techniques we describe in the book, we would see less variability and better decisions. 

Obviously we hope every policymaker and leader reads this book. As the New York Times wrote in its review of our book: “We are living in a moment of rampant polarization and distrust in the fundamental institutions that underpin civil society. Eradicating the noise that leads to random, unfair decisions will help us regain trust in one another.” 

Olivier Sibony is Professor of Strategy at HEC Paris. Previously, he spent 25 years with McKinsey & Company, where he was a Senior Partner. 

Daniel Brown is Chief Editor at HEC Paris.