Intelligence Analysis Techniques: Analysis of Competing Hypotheses

 This wasn't supposed to be an intelligence blog, but here we are... 

Today, I want to talk about a very powerful technique of intelligence analysis called "Analysis of Competing Hypotheses" (ACH). And, crucially, this isn't something that needs to stay in the halls of Langley, VA (home of the CIA) - it's something you can probably use.

Previously I talked about "all-source intelligence" - where you take intelligence from a bunch of different sources to form a more holistic picture of what's happening. We also mentioned the possibility of contradictory evidence, i.e. your HUMINT source was saying that the Hostilian (we were still calling it XYZ then) fleet was undergoing retrofit, but your GEOINT was showing that the naval yards were empty.

The point of ACH is to prevent intelligence analysts from developing "tunnel vision" - from being so focused on "proving" some presupposed "fact" that they ignore any other evidence. One famous example of this confirmation bias is how the US intelligence community handled information about the Iraqi chemical weapons program - they decided that there were two possibilities, that either Iraq had a large chemical weapons program, or they had a small one, and all of their evidence was forced to fit into one of those two possibilities. They completely ignored the third possibility, that Iraq had actually done what they had committed to and shut down those programs, with tragic results.

So, how is ACH done?

It starts with identifying your competing hypotheses. Remember that these are competing hypotheses, meaning that they should be very distinct, almost opposites from each other. ACH isn't so useful in identifying degrees of difference in two similar options.

Next, you want to gather all relevant information - your all-source intelligence.

Draw up a matrix with all of your hypotheses along the top, and all of your information items along the side. For each intelligence item, assess whether it supports (S), refutes (R), or is neutral for each hypothesis. If you have intelligence that doesn't provide any value for any of the items (it's neutral all the way across), discard it.

Assess each hypothesis on the basis of the evidence you have. Possibly counter-intuitively, you want to disprove hypotheses, not prove them. This can help you identify a "tentative conclusion" - the one with the least amount of disproving data associated with it.

See whether there's any key pieces of evidence that are crucial - this is a measure of the "sensitivity" of the conclusion. If, for example, your conclusion depends completely on a single piece of intelligence, and that item turns out to be wrong, then so are your conclusions.

Finally, report your conclusions. This doesn't just mean you present the most likely hypothesis - it means you present all of your hypotheses (that haven't been completely discounted), along with any additional lines of questioning that should be pursued to solidify your conclusions.

So let's see what this looks like in practice. Returning to our example about the Hostilian fleet, our government wants to know if they're poised to attack us. We start with two hypotheses here that are (at least for now) pretty contradictory: that either they are on their way to attack us (we'll call this hypothesis 1, or H1 for short), or that they are undergoing refit (H2). Next, we gather as much information as we can: we've already talked about the HUMINT and GEOINT, but we also have MASINT (which doesn't show unusual levels of communication or activity at their Naval headquarters) and an OSINT report from the Hostilian newspaper about their government signing a big contract with a private shipyard earlier this year. So we put all of that together into a matrix as below:

ACH Matrix example

Looking at the available data, it seems that H2 is more likely (and suggests that we need to get some GEOINT coverage of that private company's shipyards to confirm). So, based on the above, we can report to the government that it's unlikely that we will be attacked by the Hostilian Navy any time soon (although, once they're done with the refits, the situation could change).

So, how could you use this in your own life? If you've got a decision to make, you can use ACH to assess the pros and cons of each decision (and if any factors are key - remember what I said before about "sensitivity" to key data).