The Knowledge Machine: A Book Report

Michael Strevens' book, The Knowledge Machine, is one of those books you read first with your eyes, in the same way that food is first visually appealing. But it requires another slower reading, at least for me, to get more of the nuance of the dish. [1]

For the American Council on Science and Health’s readership, I want to consider one aspect of the book as it pertains to arguing and debunking “science.” In short, Strevens suggests that science has advanced because of an “iron rule” that all scientific debate be based solely on empirical, observable data. That explanation or hypothesis comes with its own ecology of assumptions and beliefs, which must, in turn, be grounded in empirical findings. Findings, where slight variations alter outcomes, necessitating exceedingly fine measurement, which changes over time. To “follow the science,” we must discard all the intellectual clutter involved in connecting the dots and deal solely with those empirical findings. 

But scientists are humans, incapable of complete objectivity. So frequently, scientists, as they mull over findings tuning their instruments and measures, and exploring theories, break the iron rule and instead consider logic, opportunity, funding, or the beauty of the approach. Strevens suggest that this is accepted because, in the formal presentation of papers, all those subjective thoughts are “sterilized” away, leaving only the empirical observations as proof.

Debunking

Very rarely do we at the American Council on Science and Health argue over the integrity of a scientific paper’s data. The only time I can recall is the hydroxychloroquine study using data from Surgisphere, data that only one of the three authors had “collected and seen.” As I look back on my writing, particularly with the “iron rule” in mind, I see that my debunking and skepticism involve issues of measurement and or disagreement with the explanatory connecting of dots. 

Arguing over measurements

Much of the debate revolves around how appropriate a given metric is in clearly identifying your variable of interest – are you counting apples, or are there a few oranges in the mix? Speaking of fruit, frequently long-term dietary studies involve a self-assessment of meals extrapolated over years. I often object to that as a valid measure because I believe our diet changes over time. The more I come to look at studies involving our health, the more I recognize that measurement of people by age, gender, and race are not so much invalid as imprecise; failing to capture the nuances – resulting in grouping apples with oranges and altering outcomes.

Of course, the all-time favorite measure to be questioned are p-values and other statistical instruments. In the last few years, the entire debate over p-values has little to do with calculating a dataset’s p-value. It has everything to do with what the p-value measures; is there a difference or not? Does the value have to be 0.01 or 0.05, or 0.001 to count as a difference? This is not an objective decision; this is subjective.

Explanation

While scientific speech is limited to what is observable, scientific thought is not. Much of the criticism of scientific studies deals with the “discussion” section when scientists relax the “iron rule” and speak with a bit of their heart and head. The deluge of scientist’s thoughts on COVID-19, found in social media and news, has dramatically extended the reach of discussion and is both widely misunderstood and misappropriated.

As Strevens points out, explanation and interpretation come with their own ecology of assumptions and accessory theories. For example, the studies of sugary beverages are predicated on the belief that reducing their intake will improve your health without considering your physical activity. The unspoken assumptions underlying a scientific theory make commentary less certain than it appears, irrespective of the eminence of the author or journal. The comments that follow our articles and the use of Twitter to “popularize” scientific papers is an empirical demonstration of subjective opinion is passed along as objective scientific fact.

Much of the seemingly contradictory findings and misinformation are nothing more than a commentary on the changing objective findings. Consider COVID-19 and packages. First, the little evidence we had found viral particles up to 48 hours after being deposited on cardboard in the laboratory, suggesting it was a source of concern. The commentary was to wear gloves, decontaminate the packages, etc. With time, objective findings in our day-to-day lives showed little viral spread. Can you remember any reports of cases of COVID-19 caused by a delivery from Amazon? The commentary shifted, no longer requiring hazmat suits to open a box. Science had not flip-flopped. It takes time to collect enough data to identify the preponderance of evidence successfully. The patience to wait for accumulated evidence to speak is non-existent in the attention-demanding world of social media and a 24-hour news cycle. 

What has changed is our explanation of those empirical findings, the stories we tell, that with time, turn out to be fiction. The news and social media are one step more removed, telling a story of what a scientific article told them. With its inevitable introduction of pundits, experts, and writers' biases, it is in the retelling that science appears to flip-flop. We mistake critique for the science.

I have only scratched the surface of The Knowledge Machine. If you are interested in science, its politicization, or its use in policy and regulation, Strevens's work will give you a much better understanding of this very human enterprise. 

[1] For those of you who want to get more of the book’s appetizer, to continue the food analogy, you can read this short piece from Aeon.