FloCon 2020

Human-driven network forensics activities (such as threat hunting and incident response) focus on identifying the source of potential network threats or other problem incidents. Analysts must sift through large amounts of network data to find forensically relevant events. Full packet payloads (called packet capture, or PCAP) have long been considered as the gold standard of forensic evidence. While full packet capture does contain all relevant forensic information, capturing and storing every packet for an extended time period is often prohibitively expensive and inefficient to analyze in bulk.

Because of these shortcomings, network analysts often turn away from full packet capture to alternative forms of forensic data. Popular alternatives include NetFlow, extended (augmented) flow, and application metadata (DPI). These alternatives provide forensic value and use significantly less disk space than full packet capture, but lack the complete packet payloads needed to fully confirm the presence of malicious activity on the network. This trade-off between the forensic value of data and the size and cost of storing it has caused analysts to seek an optimized balance between full packet payloads and other forms of forensic data.

We describe a machine learning (ML) driven approach to this storage dilemma that uses open-source tools that provide what we call intelligent packet capture. Intelligent packet capture combines incremental flow updates, streaming machine learning, and threat intelligence to classify flows and predict which packets are likely to be interesting to network analysts. Selected packets are saved to disk and combined with augmented flow meta data. This provides the best of both worlds: fewer storage demands with more forensically relevant data.

Attendees Will Learn:
Attendees will learn to build and deploy a cost-effective network forensics solution with open source tools like Argus and Dragonfly Machine Learning Engine. They will also learn how to operationalize threat intelligence feeds and apply machine learning to large-scale flow analysis.