The Rorschach Mistake
Why science embraces skepticism, and so should you
I was reading John Mandrola’s Substack Sensible Medicine this week, and a single image from one of his recent essays has not let me go. He was writing about a set of clinical trials for a cardiac procedure called left atrial appendage closure. The data is mixed. Some trials look favorable. Others, on careful reading, do not.
A colleague of his had offered a generous framing for the disagreement. He said that how you interpret these trials is like a Rorschach test.
Mandrola did not buy it. Neither do I. But the framing opens a door into something larger than any one procedure, and I want to walk through it.
You probably know the Rorschach. Ten symmetrical inkblots, designed by a Swiss psychiatrist in 1921. The cards have no inherent meaning. What you see is supposed to tell us something about you. Your fears. Your priors. Your unconscious furniture.
The metaphor is seductive. Two thoughtful physicians look at the same data and walk away with different conclusions. Therefore, the data must be ambiguous. Therefore, the disagreement is really about us, and only incidentally about the evidence.
I disagree. And I think the disagreement matters more than the trials themselves.
The Philosopher’s Trap
Let me concede the hardest version of the Rorschach view.
The philosopher Larry Laudan made an argument called pessimistic meta-induction. Most scientific theories in history have eventually proven to be wrong. Therefore, by induction, most current theories will also eventually fail. Capital-T Truth, in this view, is something we never quite reach.
I find this argument hard to refute. Phlogiston was real until it wasn’t. The four humors guided medicine for two millennia. Bloodletting killed more people than it saved. Stomach ulcers were caused by stress until a researcher swallowed a culture of Helicobacter pylori and won the Nobel Prize.
Medicine is a graveyard of confident certainties. We have today’s best guess. Nothing more. Nothing final.
Accepting all of that could tempt you to accept the Rorschach metaphor as well. If we accept that we can never truly know anything, then each of us is surely just seeing inkblots through the lens of our biases.
If you accept all of that, you might feel inclined to accept the Rorschach metaphor as well.
This is the philosopher’s trap. It moves from “we cannot know everything” to “we cannot know anything.” That move is wrong.
Degrees of Accuracy
What we lose when we accept meta-induction is the dream of certainty. What we keep is something more valuable. Degrees of accuracy.
We can know that some claims are better supported than other claims. We can know that a randomized trial with three thousand patients tells us more than a case series of forty. We can know that a pre-specified primary endpoint is more trustworthy than an endpoint that shifts after the data comes in. We can know that an industry-funded trial with a softened margin tells us less than a non-industry-funded trial with a rigorous design.
The inkblot has no signal. The trial does. The signal might be weak. Clever statistics may obscure the signal. But discipline can read the structural signal.
Newton was wrong about gravity. Einstein is probably wrong too. Yet Newton was less wrong than Aristotle, and Einstein is less wrong than Newton. The arc of science bends slowly toward greater accuracy. That is all we get. That is enough.
The question worth asking is whether we are closer than we were yesterday.
What Priors Actually Are
Buried inside the Rorschach metaphor is a deeper claim. The Rorschach metaphor suggests that priors are biased. The more prior knowledge we have, the more it contaminates our perception. The most objective reader of a new trial is the reader who knows nothing.
This is backwards. Bayes was right.
Priors are how data finds its meaning. They are the structures that let additional evidence update us. A trial arrives into a landscape of previous trials, mechanistic plausibility, biological reasoning, and lived clinical experience. The trial moves the landscape. It does not erase it.
When I encounter a new study on a procedure with a checkered evidence history, I am not staring at an inkblot. I am reading the next chapter in a book I have been following for years. Earlier chapters matter. Failed trials matter. Regulatory hesitations matter. Mechanistic plausibility matters. The new chapter either confirms the direction of the story or changes it.
A reader who treats every chapter as the first is amnesiac and calls it objectivity.
The Rorschach Confession
So when someone tells me that evidence appraisal is a Rorschach test, I think they are confessing something. They have stopped reading the book. They have stopped tracking the arc. They will call the latest chapter whatever the publishers wish them to call it.
This is intellectual surrender, and we need to reclaim our individual intellectual skepticism.
True humility goes the other way. True humility says, “I will keep getting closer to the truth without ever reaching it. I will weigh the evidence by its design. I will let strong data update me. I will let weak data leave me where I was. I will hold priors lightly enough to be moved, and firmly enough to be honest.”
Call that humility, honesty, authentic skepticism.
Call it discipline.
Call it the practice of medicine done well.
It is the opposite of staring at inkblots.
As Always,
Jordan
aka Dr. J
