Discounting Weak Evidence

One pitfall to avoid is prematurely discounting seemingly weak evidence. Weak evidence can take many forms. It could be evidence that seems very unlikely under all hypotheses. Or it could be evidence that is non-intuitive and doesn’t seem to fit what we consider “conclusive” evidence.

When evaluating evidence, it’s easy to get distracted looking for “irrefutable” evidence (more on that in an upcoming blog post). However, that’s a mistake. What’s really important is the ratio between how likely evidence is under the hypotheses.

“Rare” Evidence: The MH17 Analysis

Even when events seem highly improbable under all hypotheses, slight nuances in the likelihoods can still make a difference. If an event is rare under all hypotheses, but less rare under one of them, the evidence is relevant and impactful.

Rootclaim’s MH17 analysis highlights this phenomenon. For example, in one piece of evidence, a Russian newspaper reported on claims from a secret witness. This secret witness was quoted as saying that three Ukrainian fighter jets had flown over the area that day. According to the witness, two planes were shot down. The third fired an air-to-air missile (presumably at MH17).

A report of this kind is estimated as 5% likely under hypotheses which assume that the Ukrainian Army fired an air-to-air missile. On the other hand, this is estimated as only 1% likely under other hypotheses. That difference may look small–after all,  the occurrence is highly unlikely under all hypotheses. However, a 5% occurrence is 5 times more likely than a 1% occurrence, so this distinction is quite meaningful.

 

High vs. Low Percentages

The same marginal difference matters less when the percentages are higher versus when they are lower. That is, the difference between 1% and 5% (5x) matters more than the difference between 95% and 99% (only 1.04x) . In fact, the 1%-5% discrepancy is far more impactful than radar evidence which results in a 70% vs. 80% (1.14x) likelihood discrepancy.

 

“Unintuitive” Evidence

Popular movies and shows about crime and justice shape our perception of which evidence matters. We expect crimes to be solved using bloodstains, hair follicles, and confessions made in an oppressive police investigation room. In reality, more mundane, but unlikely, factors can also play a role. Daily schedules, weather patterns, financial records–if the evidence is more likely under one explanation than another, then it matters.

For example, in the “Serial” analysis, Adnan maintaining his innocence for 17 years is an influential piece of evidence. It may seem trivial. But when data on confessions and false convictions is incorporated, it matters. Very few guilty prisoners continue to maintain their innocence in jail, while innocent prisoners are more likely to do so. Similarly, an alleged visit to a friend (Kristi) on the day of the disappearance is much more likely if Adnan wasn’t already preoccupied with killing and burying Hae. Neither maintaining innocence nor visiting a friend fits our stereotype of decisive evidence. But in both cases, the likelihood of the evidence varies significantly based upon the hypothesis. Thus, these pieces of evidence are very relevant.

 

“Circumstantial” vs. “Direct” Evidence

Court proceedings often involve discussions of circumstantial vs. direct evidence. By definition, direct evidence points directly to something occurring (for example, eyewitness accounts). However, circumstantial evidence requires an extra step to prove the same event. For example, a receipt shows that the suspect was around the corner 10 minutes before the crime occurred. Or his clothing has stains similar to the crime scene area. These indicate a certain connection, and make it more likely that he was involved. But it is still not certain or decisive.

However, the circumstantial-direct distinction does not mean that direct evidence is always stronger. In some cases, circumstantial evidence may prove more impactful. For example, evidence X may be direct, but relatively likely under both hypotheses. Circumstantial evidence Y on the other hand may be somewhat likely under one hypothesis and very unlikely under another. Here, too, what matters is the overall relationship between the evidence and the hypotheses.

 

Weak Evidence Isn’t Really Weak

In truth, weak evidence is a misnomer. What matters is accurately assessing the likelihood of each piece of evidence given each hypothesis and the reliability of each source. It’s the differences between those evidence-hypothesis relationships that matter the most, not the absolute likelihood of any particular relationship on its own.

Effectively evaluating the likelihoods of relatively rare events and properly incorporating them into a holistic analysis is one of the strengths of Rootclaim’s Bayesian calculation. Visit Rootclaim and share your own thoughts on MH17, Serial, and other analyses, to help further refine the numbers.