The Illusion of Precision
posted 2011 by Peter Wastholm
There are three kinds of lies: Lies, Damn Lies, and Statistics. -- Benjamin Disraeli
Many years ago, while I was in the employ of a large international consulting company, I once helped a colleague out with a document he was working on. This took maybe half an hour, but ended up causing at least four different people to spend a total of at least a couple of days' worth of work trying to satisfy the corporate hunger for precise numbers, thereby illustrating the irrationality of trying to measure things too exactly.
At this and, I'm sure, most other consulting companies, keeping track of time spent on various activities was considered very important, and weekly time cards were expected. So, at the end of our brief work session, I asked my colleague how I should report the time. He gave me an activity number and, come Friday, I submitted my time card (which wasn't actually a card but a file) and forgot about the whole thing. Well, until a few days later when someone from elsewhere in the company called me and asked me why I had reported time on a one-off activity that had been concluded months earlier.
To make a long story short, this resulted in a whole lot of back-and-forth between various organizational units in different parts of the company's sprawling bureaucracy. (At this point, I'd like to point out that this was largely a well-functioning company -- but, as far as I know, every organization in history above a certain size has had a sprawling bureaucracy. I'm sure the builders of the Great Pyramid had some hellish paperwork, uh, papyruswork, to contend with.)
Now, since I bring it up here, clearly I feel there is something to learn from this incident, and I have already hinted at it. In everyday life, most people understand that close enough is, well, close enough. We let the soup simmer "a while," we add "a pinch" of salt to it, we serve it with "a few" bread rolls. In everyday life, we typically don't try to measure things with five significant digits, because it's clear to us that the effort to do so would outweigh any possible benefit. Our results may not be perfect, but they're good enough and we know what to expect from them. But in some situations, like in large organizations with sprawling bureaucracies, the effort somehow becomes invisible to to us because it's being expended by someone else. So we tend to assume that more precision is always better than less, and -- which is worse -- begin to trust numbers just because they appear to be given with an impressive level of precision.
Precision isn't merely a matter of using a large number of significant digits, offering lots of choices in opinion polls, or slotting every single half-hour of work into one of a large number of neatly labeled activities. If we insist on a higher level of precision than our instrument is capable of, we don't actually get precision, we get the illusion of precision.
Knowing how precise we can expect our data to be isn't always easy; in fact, there is an entire scientific field that studies this. It's called statistics, and it can be said to underpin most other sciences. Given the way our data was collected, statistic methods can tell us how much confidence we can put in it. We don't all need to become statisticians, of course, but it might behoove us to think about how to collect the data we base our opinions and decisions on -- or how it's collected for us. This way, we can assess to what degree we should trust opinion polls, financial information, or, for that matter, aggregated time reports across an organization. Insisting on overly precise data and relying too blindly on it may not be a good idea.
My sources are unreliable, but their information is fascinating. -- Ashleigh Brilliant