Any business enthusiast for constructive ambiguity might blanch at the uncompromisingly direct title “How To Lie With Statistics“; although they probably read this short and powerful text on their MBA course. Maybe, like me, they read it many years ago: the dated examples just add to the book’s charm (it was published in 1954).
This text was my introduction to critical thinking. For starters, here’s how to deal with crime: “Theodore Roosevelt, as president of the reform Police Board, was seriously embarrassed. He put an end to the crime wave simply by asking Steffens and Riis to lay off. It had all come about simply because the reporters, led by those two, had got into competition as to who could dig up the most burglaries and whatnot. The official police record showed no increase at all.”
The graphical presentational tricks in Darrell Huff’s text are so well-known these days that you can’t get away with them on the professional stage any more (“for that, this chart lacks schmaltz. Chop off the bottom”). But there are enough new tricks around to make any statistician hop with rage – the misuse of colour being one. Even a simple traffic-light chart may fail in front of an international audience, but colours have cultural significance. My US counterpart keeps me honest: he is colour-blind.
The worst statistical lies are the one you can’t see, because the information is missing from the report. This could be biased samples, tiny samples, insignificant variations, or unstated bases for comparison, for example. These are all traps into which the well-intentioned may fall.
There’s really only one good use of statistics in information security, and that’s to improve a critical security process that you own. Your aim may be to speed up security patching, or to eliminate bottlenecks in incident response. The statistical part is harder than it looks, as anyone who has tried to construct a security dashboard will know – even when you are trying to present a true and fair account. You just need to…
Start Two Years Ago.
… I’m afraid so. You need real data from your own organisation, not outside.
Content from our partners
External data only becomes useful once you can compare your trends with their trends. Unless you are huge, that means more than a years’ worth to get a meaningful sample size. A lack of good data plagues security metrics; with tiny samples and no historical basis for comparison, you’ll just fool yourself. Beware of trying to measure less tangible factors like annual loss expectancy. As Dan Geer said: “The data is too poor even to lie with.” Insurance companies have decades of comparable data; you don’t.
Collect the Metrics for Yourself.
It’s your own critical security process that needs to improve; you want to make the right change. Pay attention to data quality; get the real numbers, and don’t take at face value what you are told by others. By collecting only the data you need, you can be agile. Focus this effort on making change, and then repeating the measurements. This is not an opportunity to turn the activity into some kind of security analytics Big Data initiative – such initiatives need to be planned quite differently.
Follow the Scientific Method.
Using qualitative information and your intuition, create a hypothesis, then collect the data to test it. “This step takes too long because…” If the data can’t confirm or refute your prediction, don’t bother collecting it.
Talk Back to your Statistics.
Darrell Huff suggests these questions:
- Who says so? Answer: you do. Stand behind the numbers and the conclusions.
- How do they know? Answer: because your sampling methods were sound, and your data is good. Isn’t it?
- What’s missing? Answer: not much. You recorded the reliability of the metrics, the types of average used, and anywhere the basis of comparison or measurement changed. Didn’t you?
- Did somebody change the subject? Answer: no. Your definitions are consistent. Your conclusions match how the data was gathered. True?
- Does it make sense? Answer: yes. The findings may be unexpected or unwelcome, but the conclusions you drew from them are warranted. Aren’t they?
Finally, Report the Improvement you Achieved.
The only metrics that management care about are business ones, including the time and money you have saved. If you are actually asked to report security metrics, find out why. Here, I recommend Andrew Jaquith’s book (he’s no fan of traffic-light presentations either). An extract from this slim volume “How to Lie with Statistics” appears in the fat tome “The Penguin Book of Lies“.
Nowadays, that whole anthology reads like a training manual for fake news vendors, but that’s a topic for another day.
