By Jörg Abraham, Threat Intelligence Analyst & Sergey Polzunov, Software Engineer
Last month Sergey Polzunov and Jörg Abraham presented at the FIRST CTI Symposium. The topic of their talk was Evaluate or Die Trying – a methodology for qualitative evaluation of cyber threat intelligence feeds. If you didn’t make it to London, this blog post details what was shared at the event.
The operationalization of narrative-based intelligence reports – reports that describe in detail a series of events related to an intrusion or incident – is time-consuming for cyber threat intelligence (CTI) analysts.
The lack of report automation is what makes them particularly time-consuming. CTI teams need to ensure they are properly staffed and allocating enough time to make the best use of this type of reporting.
This is one of the key findings in the recent SANS CTI Survey – The Evolution of Cyber Threat Intelligence (CTI) by Rebekah Brown and Robert M. Lee (February 4, 2019).
At EclecticIQ we believe that Structured Threat Information Expression (STIX) addresses this exact problem. STIX is a key enabler for automating the acquisition, exploitation and dissemination of intelligence. In the context of exploitation, STIX greatly helps analysts to consume the narrative of intelligence reports, as it converts unstructured text into structured building blocks that can be queried.
Unfortunately, we rarely see intelligence sources that make use of the real power of STIX. All too often, we notice that threat reports append a list of IOCs, or re-package CSV files into the STIX format without any additional context.
To be clear, there are plenty of very good intelligence reports, but we think there is still a gap between the narrative in reports and the structured CTI delivered with them.
Based on these observations, we started a proof of concept to determine and assess how well a STIX feed is structured. And whether we could develop metrics that differentiate the well-structured feeds from the poorly structured.
STIX 2.0 – A standard to Compare Them All?
For our research we analyzed six feeds. Three were commercial and three were open source. We decided to use the STIX 2.0 standard as a common format for comparison.
Unfortunately, there are not many native STIX 2.0 feeds available yet. Furthermore, most sources make their data available via a JSON API, leaving it up to the consumer to interpret the data and to convert them into STIX. Hence, we had to convert existing STIX 1.2 feeds or JSONs into STIX 2.0.
We acknowledge that the process of us mapping STIX 1.2 or JSON into STIX 2.0 potentially introduces the possibility of wrongly mapping and characterizing STIX objects. However, for our goal of developing a methodology, we considered that risk to be acceptable.
Finally, we stored all ingested data in a PostgreSQL database and used a Jupyter notebook for our analysis.
A First Look at the Analyzed Data
Object variability
A very simple but effective way to assess the feed structure is to look at the variability of STIX Domain Objects in the feed. In Figure 1 and Figure 2, you will see that Source A contains a rich set of STIX entities, whereas Source B provides indicators of compromise only.
CTI consumers should ask themselves:
- Do the object types in the feed align with our production requirements?
- Is a feed balanced or is it heavily skewed to one particular object type?
- Are there custom STIX objects (outside the standard) in the feed?
An organization would first need to investigate what information the producer conveys against these criteria. Custom objects may also cause problems when ingested into security controls or a threat intelligence platform without validation.
Timeframe and Gaps
We also wanted to determine whether a feed contains enough historical data.
As seen below, Source D provides data as of January 2018, while Source E contains data going back to January 2011, albeit with a few breaks in between. The graph also allows organizations to identify significant spikes as highlighted in Source D, or trends relating to the volume of data in (Source E).
Daily Influx
When we combine the two previous graphs we can also look into the daily influx per object type. This can help organizations understand the average number of entity types ingested, to identify spikes or imbalances. Figure 4 shows that Source A ingests different object types on a daily basis.
Source C only contains STIX report entities. Strictly speaking, we do not consider these report objects to be structured intelligence. Information about actors, malware, exploit targets or victims is still hidden in a narrative inside these reports and not represented as STIX Domain Objects.
Fullness of STIX Domain Objects
Figure 5 shows the STIX Identity object that can represent individuals, groups or organizations. As shown in the image, the STIX standard defines a set of mandatory and optional fields. We wanted to know if and to what extent feed providers make use of optional fields, hence providing additional content for an entity.
Figure 6 shows that Source A and Source F contain optional fields, though none of them reaches 100 per cent fullness.
In addition, Source A contains custom fields. While custom fields are likely to contain more context about an entity, the downside is that these fields are not in the STIX standard and a consumer would first need to investigate the nature of the information stored in these fields before being able to process them.
Relationships Between STIX Domain Objects
Context also comes through relationships. Hence, we wanted to come up with graph to identify if a feed contains relationships to other STIX entities.
The histogram in Figure 7 allows organizations to analyze how entities are connected on average. Earlier in Figure 1 we showed that Source A contains different STIX entity types. The histogram for Source A now also demonstrates that the feed uses different relationship types between the objects.
Source A also contains custom relationship types indicating a relationship outside the STIX standard. As with any custom data, organizations would first need to investigate these before processing.
Source B only contains the ‘created_by_ref’ relationship type. Since this type is characterized as metadata about the entity producer and it only relates to the IOCs in this feed (see Figure 2), we did not count it as relationship in the histogram.
What´s Next?
You might ask yourself why we focus so heavily on structure in cyber threat intelligence.
We truly believe that structured threat intelligence will help organizations to automate intelligence consumption, overcoming the time-consuming barriers we mentioned earlier. But structuring intelligence will also allow organizations to measure if a CTI feed delivers on its intelligence requirements.
Imagine an organization that obtains information from a well-structured cyber threat intelligence feed as shown in Figure 8. Provided the consumer has well-defined intelligence requirements as seen on the left side of the image, it is fairly simple to map them to the intelligence delivered in the feed (right box).
The organization can calculate what percentage of information within a feed relates to its own intelligence requirements. If there is little to no overlap, why should that organization consume the feed in the first place?
Sergey and I demonstrated that this exemplary assessment can already be done today. We took one of the well-structured feeds from our study and calculated how much of the provided intelligence in that feed relates to an industry sector (Figure 9) or to what extent a feed covers malware families of interest (Figure 10).
For the same reasons, we believe that intelligence providers that offer a properly structured STIX feed can also benefit. Like CTI consumers, producers can test and demonstrate how their intelligence product relates to customer requirements.
By referencing the key findings from the SANS report from the beginning, consumers will value structured intelligence that follows a clear data model with minimal room for interpretation that eventually reduces the time needed to consume the intelligence. In fact, we believe that a well-written and structured STIX feed can become a unique differentiator for CTI providers.
We hope this blog gives you a glimpse how a thorough analysis of STIX feeds can aid intelligence consumers and producers to evaluate and eventually improve their products. If you want to learn more about our work, you can review our presentation on the FIRST Program website or reach out to us on Twitter.
We hope you enjoyed this post. Subscribe to our blog for more interesting reads on Cyber Threat Intelligence or check out our resource section for whitepapers, threat analysis reports and more.
