In the first part of this blog series we talked about some initial hurdles that the intelligence community has in structuring Analysis of Competing Hypotheses (ACH) in a consistent way.
In this follow-up post we will address how analysts first thought about tackling this problem. Then in the final part of this three-part series, we will discuss a potential and proposed solution.
Introduction
We previously looked at what ACH means and how it can support not only security analysts, but also consumers and producers of cyber threat intelligence (CTI).
The main problem is that there is currently no way to structure the process of testing evidence against a set of hypotheses (H1, H2, H3…). Producers of intelligence often create multiple, competing hypotheses around a given threat to identify the strongest hypothesis (H1) most supported by the evidence available.
One problem we found is that there is no way to ingest Digital Shadows data in STIX that will allow the consumer to see all the hypotheses being tested and to see the evidence used to support each one.
Using the STIX data model
The STIX model (even 2.x) is very flexible and offers much room for individual interpretation. This means analysts working on the same team can easily create multiple and different interpretations of the same data.
This is a general problem, but it is particularly difficult in an environment where an analyst may work across multiple feeds and sources, each with its own understanding of how the data should be structured.
It’s for this reason that EclecticIQ Fusion Center analysts have decided to adhere to an agreed-upon data model to scale across many feeds/sources.
If we consider the issue of being unable to structure ACH, we see that there are not only limitations in the flexibility and interpretation of STIX, but also limitations in the lack of options available to structure the process of conducting ACH.
The need to structure ACH comes from the need to ask pertinent questions about our data. For example, some of the limitations include:
- The inability to structure information from vendors testing competing hypotheses
- The inability to identify which hypothesis was determined to score the strongest from the set of hypotheses being tested
- The inability to separate an alternate/proposed reality from a confirmed reality
The need to structure this hypothesis-testing process is driven by several factors, or rather questions we would like to be able to ask of our data:
- How can we identify patterns in authors’ assessments over time? (i.e. show me what the world looks like when PROVIDER A’s hypotheses are true.)
- How can we identify evidence that is consistently being used to support multiple hypotheses over time? (i.e. exploiting CVE-X-X as an initial attack vector is used as evidence to support more than one hypothesis.)
- How can we identify patterns or trends when undertaking threat actor attribution? (i.e. show me all hypotheses that support attribution of a Russian Advanced Persistent Threat or ‘APT’.)
- How can we measure the predictability or the sophistication of a threat actor? (i.e. more than 80 per cent of our hypotheses on this threat actor were correct; or less than five per cent were correct, so clearly this threat actor changes tactics and is unpredictable.)
What the STIX 2.1 Opinion Object permits
The STIX 2.1 Opinion Object is the entity that gets the closest to being something that can be used to structure hypothesis testing.
The Opinion Object is able to express ‘agreements’ or ‘disagreements’ about other STIX entities or relationships. However, it is clear that it is not meant to be expanded beyond its current form based on the below limitations:
- The current Opinion Object specification does not address how the community should use and apply the Opinion Object. As we have discovered, while STIX is very flexible, analysts from the same team may end up modelling the same dataset or intrusion set differently.
- One of the largest caveats of the Opinion Object is that sharing communities are still encouraged to provide clear guidelines to their constituents regarding best practice for the use of the Opinion Object. This means there is no fundamental agreement on when and how to best use this Object.
- The Opinion Object does not apply any additional structure beyond a free-text ‘explanation’ as to why an author has an opinion in the first place.
- There is no way to consistently track or see patterns in ‘explanations’ for Opinions over time
It is clear that the STIX 2.1 Opinion Object cannot be used to structure hypothesis testing. It’s also clear that there is a need for a new object that would be able to do this.
The Opinion Object introduces a free-text explanation for the reason why the consumer/producer has this opinion. There is a need for objects that rely on a free-text field. However, for the purpose of structuring ACH, we felt like there was a need for a new object.
The Opinion Object -Limitations and the argument for a new entity
The Opinion Object focuses on validating intelligence from various sources, but only in a free-text, non-evidence-based fashion.
Being able to have an entity with the sole purpose of relaying an analysts’ thoughts and opinions about an entity or a relationship in a free-text fashion is useful. But that is the minimum needed.
In addition to validating a source of commercial data, it’s clear that there is a need to be able to structure evidence in support of a hypothesis and to be able to structure this process of conducting ACH.
Producers and consumers of CTI practice hypothesis testing to seek attribution of a threat actor, or to better understand an emerging threat. At the moment, STIX has no way to structure this thought process or the evidence that supports one hypothesis over others.
In part 3 of this blog series we will explore what characteristics makes a new entity viable for representing ACH.
We hope you enjoyed this post. Follow us here for more interesting reads on Cyber Threat Intelligence or check out our resource section for whitepapers, threat analysis reports and more.
