There’s a common notion floating around, due to Karl Popper, that scientific theories are characterized by the fact that they are falsifiable.  The idea is that it is never possible to verify a scientific theory (i.e. the sun always comes up) because one day it might not happen.  But it is possible that the sun might not come up some day, and then the theory is falsified.  It must then be rejected, and replaced with something more complicated.

Now, let me confess right away that I have not gotten this idea by reading any of Popper’s writings.  It is an idea which has been popularized in the scientific community.  You see, everyone knows what Popper said without having read any of it ourselves.  It could be that if I actually read Popper’s books, my idea of what he said would be falsified.  So let me confine myself in this post to discussing Popperism as commonly understood.

If a theory is unfalsifiable (that is, if no experiment you could possibly perform would rule it out, then according to Popperism it is not a scientific theory.  Among those who subscribe to Scientism, this is usually assumed to be A BAD THING™.  (The way some people talk, if a theory is unfalsifiable, that means it is false!)

People often characterize bogus pseudoscientific ideas as unfalsifiable, because of the tendency of people who believe in them not to subject them to rigorous scrutiny.  But this is clearly an oversimplification.  True, there is such a thing as mystical Woo-Woo from which no definable predictions can be made, either because the ideas are not precise enough or because they don’t relate to any actually observable phenomena.  But many psuedoscientific ideas, such as homeopathy, reflexology, or astrology, can be tested experimentally, it’s just that the people who believe in them don’t like the results when people do!)  I’ve heard people refer to Young Earth Creationism (YEC) as unfalsifiable.  I think their reasoning must be the following:

1. YEC is unscientific and wrong.

2. I’ve been taught that when ideas are unscientific, the reason is because they are unfalsifiable.

3. Therefore, YEC is unfalsifiable.

In fact, though, the real problem with YEC is that it IS falsifiable, and in fact has been falsified many time over. If the universe were created about 6,000 years ago and we have to get all of the layers of fossils and rock from a single planet-wide Flood about 4,500 years ago, then there are a gazillion problems with observation.  It contradicts the results of almost every branch of science which tells us anything about the past.  (Adding bizarre extra ideas, like God created the earth with fossils in it in order to trick us into believing in evolution, may make YEC unfalsifiable, but it might be better to characterize this as pigheaded refusal to accept reasonable falsification.)

[Fun fact: if you interpret all of the genealogies in Genesis as being literal, with no gaps—which of course I don’t—then it follows that when Abraham was born, all of his patrilineal ancestors were still alive, back to the tenth generation (Noah)!  (This is using the Masoretic Hebrew text that omits Cainan, who is included in the Septuagint Genesis and Luke.)]

All right, digression over.

Clearly there is something right about the idea that theories ought to be falsifiable, yet not confirmable with certainty.  Major scientific theories usually deal with generalities: they make predictions for a large (perhaps infinite) number of different situations.  Normally, it is not possible to verify them in all respects, because even if it works well in many cases, it could always be an approximation to something else.

On the other hand, I think there are some scientific ideas which are verifiable but not falsifiable.  Here’s an example:

Ring Hypothesis: Somewhere in this universe or another, there exists a planet with a ring around it.

I submit to you that: 1) our observation of Saturn verifies the Ring Hypothesis, 2) when scientists verify a proposition by looking through a scientific instrument, that counts as Science, and 3) no possible observation could have falsified the Ring Hypothesis.  (Even restricting to the Milky Way, eliminating planets with rings would be a tall order, impossible with current technology.)  Therefore, there are scientific propositions which are verifiable but not falsifiable.

On the other hand, even if an experiment “falsifies” a theory, it could be that the experiment rather than the theory is wrong. As Einstein once said “Never accept an experiment until it is confirmed by theory”.  This witticism may seem to turn science on its head, but nevertheless it has a bit of truth to it.  A while back, there was an experimental observation which seemed to suggest that neutrinos travel faster than light.  Soon there were many papers on the arxiv trying to explain the anomaly.  But it turned out, not surprisingly, that there was an error in the measuring devices.  Usually, when a well-tested theory is in conflict with an experiment, and the anomaly has no particularly good theoretical explanation, it is the experiment which is wrong.  Not always, but usually.

What this means is that we need a more flexible set of ideas in order to discuss falsification and verification.  In particular, we ought to accept that falsification and verification can come in degrees—observations can make an idea more or less probable, without reducing the probability to exactly 0 or 1.  The accumulation of enough experimental data against a theory should make you reject it, but it may be able to withstand one or two anomalous measurements.

The quick answer is that one ought to use Bayes’ Theorem instead.  This is a general rule for updating beliefs, taking into account both our prior expectations and observation.  This goes not just for Science, but also for everything else.  The only thing that makes Science special is that, due to a number of special circumstances, the process of testing through observation is particularly easy to do.

Even though falsification is not the best way to think about Science, it still works pretty well in many cases.  In a later post, I hope to explain the connection between Bayes’ Theorem and falsification.  Usually we should expect good theories of the universe to be falsifiable, but in certain situations they don’t have to be.  Bayes’ Theorem can be used to understand both the general rule, and why there are exceptions.

Today I’d like to talk about Bayes’ Theorem, especially since it’s come up in the comments section several times.  It’s named after St. Thomas Bayes (rhymes with “phase”).  It can be used as a general framework for evaluating the probability of some hypothesis about the world, given some evidence, and your background assumptions about the world.

Let me illustrate it with a specific and very non-original example.  The police find the body of someone who was murdered!  They find DNA evidence on the murder weapon.  So they analyze the DNA and compare it to their list of suspects.  They have a huge computer database containing 100,000 people who have previously had run-ins with the law.  They find a match!  Let’s say that the DNA test only gives a false positive one out of every million (1,000,000) times.

So the prosecutor hauls the suspect into court.  He stands up in front of the jury.  “There’s only a one in a million chance that the test is wrong!” he thunders, “so he’s guilty beyond a reasonable doubt; you must convict.”

The problem here—colloquially known as the prosecutor’s fallacy—is a misuse of the concept of conditional probability, that is, the probability that something is true given something else.  We write the conditional probability as \(P(A\,|\,B)\), the probability that \(A\) is true if it turns out that \(B\) is true.  It turns out that \(P(A\,|\,B)\) is not the same thing in general as \(P(B\,|\,A)\).

When we say that the rate of false positives is 1 in a million, we mean that $$P(\mathrm{DNA\,match}\,|\,\mathrm{innocent}) = .000001$$(note that I’m writing probabilities as numbers between 0 and 1, rather than as percentages between 0 and 100).  However, the probability of guilt given a match is not the same concept: $$P(\mathrm{innocent}\,|\,\mathrm{DNA\,match}) \neq .000001.$$The reason for this error is easy to see.  The police database contains 100,000 names, which is 10% of a million.   That means that even if all 100,000 people are innocent, the odds are still nearly equal to .1 that some poor sucker on the list is going to have a false positive (it’s slightly less than .1 actually, because sometimes there are multiple false positives, but I’m going to ignore this since it’s a small correction.)

Suppose that there’s a .5 chance that the guilty person is on the list, and a .5 chance that he isn’t.  Then prior to doing the DNA test, the probability of a person on the list being guilty is only 1 : 200,000.  The positive DNA test makes that person’s guilt a million times more likely, but this only increases the odds to 1,000,000 : 200,000 or 5 : 1.  So the suspect is only guilty with 5/6 probability.  That’s not beyond a reasonable doubt.  (And that’s before considering the possibility of identical twins and other close relatives…)

Things would have been quite different if the police had any other specific evidence that the suspect is guilty.  For example, suppose that the suspect was seen near the scene of the crime 45 minutes before it was committed.  Or suppose that the suspect was the murder victim’s boyfriend.  Suppose that the prior odds of such a person doing the murder rises to 1 : 100.  That’s weak circumstantial evidence.  But in conjunction with the DNA test, the ratio becomes 1,000,000 : 100, which corresponds to a .9999 probability of guilt.  Intuitively, we think that the circumstantial evidence is weak because it could easily be compatible with innocence.  But if it has the effect of putting the person into a much smaller pool of potential suspects, then in fact it raises the probability of guilt by many orders of magnitude.  Then the DNA evidence clinches the case.

So you have to be careful when using conditional probabilities.  Fortunately, there’s a general rule for how to do it.  It’s called Bayes’ Theorem, and I’ve already used it implicitly in the example above.  It’s a basic result of probability theory which goes like this: $$P(H\,|\,E) = \frac{P(H)P(E\,|\,H)}{P(E)}.$$The way we read this, is that if we want to know the probability of some hypothesis \(H\) given some evidence \(E\) which we just observed, we start by asking what was the prior probability \(P(H)\) of the hypothesis before taking data.  Then we ask what is the likelihood \(P(E\,|\,H)\), if the hypothesis \(H\) were true, we’d see the evidence \(E\) that we did.  We multiply these two numbers together.

Finally, we divide by the probability \(P(E)\) of observing that evidence \(E\).  This just ensures that the probabilities all add up to 1.  The rule may seem a little simpler if you think in terms of proability ratios for a complete set of mutually exclusive rival hypotheses \((H_1,\,H_2\,H_3…)\) for explaining the same evidence \(E\).  The prior probabilities \(P(H_1) + P(H_2) + P(H_3)\ldots\) all add up to 1.  \(P(E\,|\,H)\) is a number between 0 and 1 which lowers the probability of hypotheses depending on how likely they were to predict \(E\).  If \(H_n\) says that \(E\) is certain, the probability remains the same; if \(H_n\) says that \(E\) is impossible, it lowers the probability of \(H_n\) to 0; otherwise it is somewhere inbetween.  The resulting probabilities add up to less than 1.  \(P(E)\) is just the number you have to divide by to make everything add up to 1 again.

If you’re comparing two rival hypotheses, \(P(E)\) doesn’t matter for calculating their relative odds, since it’s the same for both of them.  It’s easiest to just compare the probability ratios of the rival hypotheses, because then you don’t have to figure out what \(P(E)\) is.  You can always figure it out at the end by requiring everything to add up to 1.

For example, let’s say that you have a coin, and you know it’s either fair (\(H_1\)), or a double-header \(H_2\).  Double-headed coins are a lot rarer than regular coins, so maybe you’ll start out thinking that the odds are 1000 : 1 that it’s fair (i.e. \(P(H_2) = 1/1,001\)).  You flip it and get heads.  This is twice as likely if it’s a double-header, so the odds ratio drops down to 500 : 1 (i.e. \(P(H_2) = 1/501\)).  A second heads will make it 250 : 1, and a third will make it 125 : 1 (i.e. \(P(H_2) = 1/126\)).  But then you flip a tails and it becomes 1 : 0.

If that’s still too complicated, here’s an even easier way to think about Bayes’ Theorem.  Suppose we imagine making a list of every way that the universe could possibly be.  (Obviously we could never really do this, but at least in some cases we can list every possibility we actually care about, for some particular purpose.)  Each of us has a prior, which tells us how unlikely each possibility is (essentially, this is a measure of how surprised you’d be if that possibility turned out to be true).  Now we learn the results of some measurement \(E\).  Since a complete description of the universe should include what \(E\) is, the likelihood of measuring \(E\) has to be either 0 or 1.  Now we simply eliminate all of the possibilities that we’ve ruled out, and rescale the probabilities of all the other possibilities so that the odds add to 1.  That’s equivalent to Bayes’ Theorem.

I would have liked to discuss the philosophical aspects of the Bayesian attitude towards probability theory, but this post is already too long without it!  Some other time, maybe.  In the meantime, try this old discussion here.

In my Pillars of Science series, I enumerated six aspects of Science that help explain why it works so well.

It should be clear from my analysis that the characteristics of Science are quite flexible.  All of the criteria are matters of degree, so that they are met more strongly by some fields of study than by others.  Because of this fuzziness, we should expect to find borderline sciences, such as Economics, Anthropology, Psychology, and other social sciences.  It is both futile and unnecessary to try to come up with a criterion to draw an exact line between science and non-science.  In other words, the question of what counts as Science cannot itself be resolved with scientific precision, and is therefore not a scientific question.

This doesn’t bother me too much because my parents are linguists.  So when I was growing up, they made sure I was aware that concepts are defined by their centers, not their boundaries.  For example, if I say the word “chair”, then what pops into your mind is a thing with four legs at the dinner table.  You might admit under interrogation that a “beanbag chair” is also a chair, but it’s hardly the first thing you’ll think of.  Concepts can be useful even when they’re a bit fuzzy at their boundaries.

Despite their flexibility, the criteria are sufficiently strict that many things don’t qualify.  I don’t just mean pseudo-sciences such as astrology or homeopathic medicine, but genuine evidence-based fields of knowledge (“sciences” in the archaic sense of the word) which aren’t scientific in the modern sense, because they only satisfy some of the criteria.

For example, History and and Courts of Law, despite their empirical character, deal mostly with unique and unrepeatable events.  So they fail the repeatability prong of Pillar I.  Both of these fields are based primarily on testimony of witnesses, although Law Court fact-finding has much stricter rules about admissibility of evidence.  Since much of their subject matter can’t be defined with quantitative precision, they don’t do terribly well on Pillar IV either.  Academics in History do have a truth-seeking community similar in kind to the Sciences.  But in Law Courts, the role of ethics, community, and authority is completely different.

This does not mean that these fields should be held in contempt; their methods are sometimes capable of establishing specific facts with a very high degree of certainty, “beyond a reasonable doubt” as the saying goes.  They simply lack the particular methodology of science, which has a proven track record of almost routinely proving astonishing facts about the world, to a degree that ends rational opposition.  If you try to increase certainty by imposing a “scientific” approach on a subject that isn’t suited for it, you risk generating a pseudo-science which jingles the jargon of science while missing its core value: self-correction through rigorous testing of ideas.

Philosophy is nonscientific for a different reason than the empirical humanities.  While many philosophers strongly value elegance and precision of ideas, typical disputes between philosophers are not very amenable to empirical testing.  That doesn’t mean that observation plays no role.  But the way philosophers typically make arguments, they also rely on controversial background assumptions, which can’t be definitively settled just by looking at the world.

If, despite the potential for controversy, the argument for the position is sufficiently convincing, this can still establish the philosophical position with great certainty.  In fact, unless the skeptical thesis that no knowledge is reliable could be refuted with near certainty, the result would be that no field of inquiry could produce near certainty.  This potential for certainty does not change the fact that Philosophy operates by a different methodology, which on average does not resolve controversies as easily as the methods of Science or even History do.

For this reason a philosophical thesis based on Science will usually have the degree of certainty associated with Philosophy, not that associated with Science.  A chain of reasoning is only as strong as its weakest link.  So a philosophical argument based on Science should not necessarily trump, e.g. a strong historical argument, simply because Science is normally more reliable than History.

So how do we fit ideas from different fields together?  In a future post, I’ll discuss Bayes’ Theorem, which is a flexible way to think about all different kinds of evidence-based reasoning, without making specific assumptions about the sorts of evidence we can include.