This statement, that extraordinary claims require extraordinary evidence, is (I believe correctly) attributed to Carl Sagan, and has become a guiding principle of skeptical philosophy. It’s little more than common sense, really, but is worth repeating. It has become for skeptics an indispensable shield against outlandish claims. For true believers it is an annoyance, and they have attacked it relentlessly.

The principle is based upon two premises: that we know stuff and that not all evidence is created equal.

It is a simple fact of history that we stand atop centuries of carefully accumulated scientific knowledge. There is a core of basic scientific knowledge that most people now just take for granted – for example that the development of biological organisms is guided by information which is stored in DNA within each cell. Most people do not even realize that prior to Mendel the prevailing belief was that templates, rather than digital information, guided development. Some facts in science are so well established that, as Stephen J. Gould noted, it would be perverse to withhold provisional assent. In other words, we know stuff.

This fact is often dismissed by the non sequitur that we don’t know everything. True – we do not, and never will, know “everything.” But not knowing everything is not the equivalent of knowing nothing. The fund of human scientific knowledge is significant. It cannot reasonably be ignored or dismissed out of hand, for it stands atop a mountain of evidence. For this reason, when a new idea or claim surfaces it is reasonable to see how the new claim stacks up against what is already well established – or at least it would be folly not to do so. If the established body of scientific knowledge has any validity, then a new idea that is compatible with existing knowledge is more likely to be true than a new idea that contradicts established knowledge. This is precisely what we mean by extraordinary.

A proponent of the alien hypothesis of UFO’s once quipped at me that “evidence is evidence.” How profoundly naïve. The eyewitness testimony of a drunkard, dimly remembered years after the event, can be considered “evidence.” So can the physical remains of a crashed ship. It should be obvious, however, that the former is almost worthless while the latter is highly compelling.

There is, in fact, a hierarchy of evidence. That which is objective, verifiable, and repeatable is more valuable than that which is subjective, anecdotal, and ephemeral. In certain applied sciences, like medicine, it is even useful to classify evidence by quality – a discipline called evidence-based medicine. EBM literature has developed a classification scheme for clinical evidence, with randomized, controlled, double-blind studies at the top, and anecdotal case reports at the bottom.

There needs to be a hierarchy of evidence because bodies of evidence are often conflicting. The job of scientists would be easier if every study of a phenomenon lead to the same conclusion, but it doesn’t. It is more the rule than the exception that evidence points in different directions, even to mutually exclusive conclusions. Weighing the evidence is a reasonable method for figuring out which conclusion is most likely to be correct, at least pending further evidence.

I don’t think anyone can reasonably differ with the statement that if I have 100 well designed and executed class 1 clinical trials showing that a drug is safe and effective in treating a certain illness, that a single anecdotal case of someone who did not respond is not sufficient evidence to overturn conclusions based upon the 100 trials.

Getting back to extraordinary claims, what this means practically is that a claim contradicts a large body of evidence. In order to accept the new claim it should have evidence at least on a par with the evidence that suggests it is not true. So if a pile of evidence suggests that a claim is extraordinary, then an equal pile of evidence would be required to combat this and support a claim as probably true. Extraordinary claims require extraordinary evidence. When you get down to it, this is just math.

The Extraordinary Mr. Wu

It boggles the mind that true believers could have a problem with this basic rule of common sense, but they often do. In fact they must if they are going to maintain belief in a claim that is not supported by a fair assessment of all available evidence.

Winston Wu, whose attacks against skepticism I have been dismantling this week, has several objections, most are straw men. For example, he criticizes skeptics by writing, “Now it would help if the skeptics who proclaim this argument specify what they would accept as extraordinary evidence. Otherwise, arbitrarily stating this argument gives one an out no matter what evidence is shown.” This is true – any principle can be abused and taken to an extreme. Skepticism can become denial. But this is not an argument against the principle itself, and Wu has not established that skeptics in general, or in any particular, abuse this principle.

The argument from Final Consequences

Wu, with characteristic misdirection and fallacious reasoning, next argues that this rule, while generally good, is “fallible” because it is possible for something to exist without leaving behind evidence, or the evidence may just not have been found yet, or it may be subject to interpretation. He states, “For example, planes, radio waves, electromagnetism, and light move around without leaving ‘hard evidence’ yet they exist.” Wu is trying to make it seem as if skeptics would deny the existence of light. These examples are absurd because they are all physically detectable.

But the deeper logical fallacy here is that things that exist must be detectable and provable by science. If they cannot be proven, then there is something wrong with the science. This is the logical fallacy known as the argument from final consequences. There is nothing about nature that states that we must be able to figure it out or prove it exists.

Wu seems to be talking about what is metaphysically True. But this rule, as with all scientific thinking and methods, is not about what is True, but what we know. How can we know scientifically that something is probably true? We require logic and evidence. If the evidence is not there, for whatever reason, then we cannot know about it – even if it’s True.

I suspect Wu’s fallacy here betrays his backward thinking. He strongly believes in certain paranormal claims and is frustrated by science’s inability to prove them. His arguments begin with the premise that certain paranormal claims are true. He states, “It is possible for something to exist yet the evidence for it hasn’t been found or understood yet, which is the case for almost every discovery in history from fire and wheels to gunpowder and gravity, to planets, atoms and electromagnetism.” He fails to recognize that he is looking back and those ideas that stood the test of time. But while atoms and electromagnetism were being discovered, there were countless notions that turned out not to be true and were discarded. He is assuming that ESP is like electromagnetism, but perhaps it is like the ether. How do we know? That’s the real question.

It’s all subjective

Wu takes a very postmodernist stance – that “extraordinary” is subjective and a matter of perspective. He seems to be taking a colloquial definition of the word as meaning – what I experience in everyday life, rather than the relevant operational definition of – contradicted by established scientific evidence.

For example he states that to fish the notion of land is extraordinary, but that doesn’t mean land doesn’t exist. He also states that to Asians the life force known as chi is an everyday experience, just like gravity. So it is not extraordinary to them. Wu seems not to realize that he is essentially claiming the cultural interpretation of subjective experience as scientific evidence. Gravity can be measured and predicted; chi cannot – the existence of chi is an extraordinary claim.

Fuzzy definition

Next Wu commits the false continuum logical fallacy – that there is a continuum from ordinary to extraordinary, and the definition is fuzzy, therefore the distinction is not meaningful. Sure, knowledge of science and good judgment are required to decide how extraordinary a claim is, but meaningful evidence-based judgments can be made.

Wu discusses at length what he believes is extraordinary evidence of UFO’s, ESP, spirits, and mystical experiences. Discussing each of these topics is beyond the scope of this blog entry, and ultimately it is irrelevant to the point. I will just say that his evidence is all old and has thoroughly been refuted elsewhere. Also, that people can disagree about what is extraordinary is irrelevant – people can disagree about everything in science. It says nothing about the utility of this principle in assessing evidence and claims.

Wu does make one statement I would like to address, however. He writes, “For hard nosed skeptics though, even good evidence will not be enough, since their mentality is to debunk rather than to discover and learn. You see, even if I had a piece of a crashed flying saucer and showed it to them, they would just say that it is probably just a piece of top secret military aircraft that we don’t know about yet. They would want the full saucer itself to be convinced. Then if I found a whole saucer and showed it to them, that would still not be enough because then they could say that there is no proof that the saucer is extraterrestrial in origin and that it could just be a secret type of aircraft invented by the military.”

I see that Wu wanted to ad “ad hominem” to his repertoire of logical fallacies. He is simply characterizing all skeptics as hopeless deniers. He outrageously claims, without evidence or justification, that no amount of evidence could ever convince a skeptic, and uses this as a way of dismissing skepticism in the face of decidedly lower quality and amounts of evidence.

It is also further evidence that he is starting with desired conclusions and working backwards. What he really should say is, “If I had an artifact that appears to be (or is alleged to be) a piece of an alien spacecraft…” In this scenario historically what has actually happened is that true believers began with the conclusion that the artifact is alien and then cherry picked evidence or engaged in mystery mongering to support their conclusion. Meanwhile, skeptical scientists examined the evidence with a characteristic combination of appropriate skepticism (extraordinary claims…) and openness, and conducted often simple examinations or experiments to determine that the alleged artifact was either a hoax or a mundane object. This is true of crop circles, of alleged landing sites, and of alien implants. So far no one has produced a demonstrably alien artifact – so Wu cannot claim to know how skeptics will respond when such occurs.

A Matter of Perspective

Next Wu claims that “extraordinary evidence” is subjective and just a matter of perspective, just as “extraordinary claim” is. This is just the “evidence is evidence” defense. He specifically equates a subjective personal experience with extraordinary evidence. The experience may have been extraordinary, but it is still subjective, and therefore not reliable as scientific evidence. This is a point that I find is often at the crux of differences between believers and skeptics. Believers are compelled by subjective evidence, and skeptics are wary of it.

I think skeptics have more than proven their case. Subjective testimony is unreliable because: people lie, memory is faulty, perception is subject to illusion, and there is a host of psychological factors that influence our experience and memory of events. All of this is well documented in the psychological literature. Also, there are countless historical events that prove these principles. One excellent example is the airship craze of the late 19th century, as documented by Robert Bartholomew.

Wu tries to make a separate point, but is really just restated his prior premise, when he writes: “Someone who believes that paranormal events are impossible is obviously going to need a lot more proof than someone who believes that they are possible and normal. However, just because miracles, ESP, sightings of apparitions, or OBE‚s haven’t happened to skeptics doesn’t mean they haven’t happened to others.”

Again he is simply mistaking subjective experience for scientific evidence. Skeptics consider paranormal claims extraordinary not because they have not experienced them but because there is no credible scientific evidence to support their existence and they contradict our current scientific models of how the world works.

In fact I have personally had many of the types of experiences that paranormal believers cite as evidence for the paranormal. I have had waking dreams (hypnagogia) in which I have sensed an alien presence, was paralyzed, and on one occasional floated across the room. My subjective experience does not make claims of alien abductions less than extraordinary, especially since such experiences are known to be a neurological (and not paranormal) phenomenon.

Wu writes: “In order for one to know what is impossible or improbable, one would have to be an all knowing creator of the universe who possesses every knowledge that there is.” Wrong again. As I stated above, one does not have to know everything in order to know anything. What Wu is essentially saying is that unless we have god-like omniscience, we effectively know nothing, and every hypothesis, no matter how apparently bizarre or contradictory, should be considered equally likely.

Skepticism is anti-progress

Wu concludes with the very tired anti-skeptical argument that skeptics impede scientific progress by supporting the status quo and resisting change despite overwhelming evidence. This is what I call the Galileo syndrome, as the great scientist’s name is often invoked to express this notion. I always find this claim to be incredibly historically naïve. Sure, there have been those in science who have resisted change – people are people. But as an institution science is more open to change than any other human endeavor. Science historian James Burke observed that science is institutionalized change. Also, as the pace of scientific progress quickens, scientists are increasingly comfortable and used to change, as there is less time to become settled with any idea. Further, new young scientists coming up and hoping to make a career look for new discoveries to make, ideas to overturn – they look to make change.

But again the paranormal proponents are starting with their conclusion, that paranormal phenomena are real, and therefore any resistance to paranormal claims must be due to a generic resistance to change. They seem to have a view of science as consisting of new ideas that are eventually validated, once resistance is removed. They either forget or do not know that science consists much more of false ideas that are disproved and discarded. Discarding false ideas is essential to science, it is like carving a statue – you can only reveal the figure underneath by taking away what does not belong.

Wu writes, “Therefore, rules should be established to clarify whether a competing theory is promising enough to warrant further research so that when those rules are satisfied, excuses can’t be used to try to dismiss the evidence off hand.”

We have such rules. A very important one is that extraordinary claims require extraordinary evidence.