Trials and tribulations of evidence based decision-making

Take Back Your Pregnancy by Emily Oster.

The key to good decision making is evaluating the available information—the data—and combining it with your own estimates of pluses and minuses. As an economist, I do this every day. It turns out, however, that this kind of training isn't really done much in medical schools. Medical school tends to focus much more, appropriately, on the mechanics of being a doctor. When I asked my doctor about drinking wine, she said that one or two glasses a week was "probably fine." But "probably fine" isn't a number. In search of real answers, I combed through hundreds of studies—the ones that the recommendations were based on—to get to the good data. This is where another part of my training as an economist came in: I knew enough to read the numbers correctly. What I found was surprising. The key problem lies in separating correlation from causation. The claim that you should stop having coffee while pregnant, for instance, is based on causal reasoning: If you change nothing else, you'll be less likely to have a miscarriage if you drink less coffee. But what we see in the data is only a correlation—the women who drink coffee are more likely to miscarry. There are also many other differences between women who drink coffee and those who don't, differences that could themselves be responsible for the differences in miscarriage rates.

Well worth a read. Her experience is not dissimilar to that of the science writer Stephen Jay Gould when he was diagnosed with an uncommon malignant cancer. He documented his exploration of the facts and statistics of his diagnosis (which was a median survival time of eight months from diagnosis) in an article, The Median Isn't the Message. Understanding what are the real facts is a task in itself. On a separate note, Oster alludes to decision-making being the evaluation of available information. That is part of it but I would suggest it is actually the iterative balance of four activities.

1) Intentions - What are your goals, how will you measure them, what are the parameters that cannot be exceeded, and how will you know when you are successful?

2) Evidence - What does the evidence actually say?

3) Estimates - For the critical knowledge needed to make the decision and which is not available, what are the best available estimates and what risks are associated with those estimates?

4) Forecasts - Once causal analysis is completed, what are the actions necessary to achieve the desired outcomes and what are the forecasts of both required resources and effort as well as forecasts of outcomes?

Intent, Evidence, Estimates, Forecasts - that sounds better and more accurate.