Creating a Security Data Platform with Snowflake and Hunters.AI
Feb 20, 2020 | 5 Min Read
Author: Omer Singer
How to Use Snowflake, Snowflake Technology
Cybersecurity organizations experience a lot of uncertainty, but there’s one thing they know for sure: Log data volumes are growing fast. The explosion of security data strains budgets and causes visibility gaps. Without a full picture of their environment, defenders can’t effectively detect and remediate threats.
At Snowflake, we hear from customers that threat detection needs a cloud-scale solution. We believe that Snowflake Cloud Data Platform is the best data platform in the world and that Snowflake is uniquely effective for threat detection at scale. Specifically, Snowflake’s separation of storage and compute means that security teams can collect and store huge amounts of log data, but they pay based on the analytics they run. As a result, security teams don’t have to choose which data is important enough to collect.
Also, Snowflake’s seamless support for semi-structured data (such as the JSON format of most modern log sources) is a must-have for threat detection and response. Many customers now collect security data into their enterprise data platform and use it for workloads ranging from metrics reporting to vulnerability management and incident response.
Many customers are daunted by the effort required to collect security data and extract insights. They tell us that they want more capabilities out of the box so their in-house security teams can focus on the unique security requirements of their company..
Recognizing that Snowflake is not a cybersecurity company (our engineers are too obsessed with extremely fast queries), my main role in developing our cybersecurity strategy is to find the best vendors in the world that can serve customers on top of Snowflake. The vision is for everyone to build a complete source of truth for their security data in Snowflake, and then share it securely with vendors that deliver analytics and automation.
That’s where Hunters.AI comes in. Hunters.AI has built an automated threat detection solution that correlates multiple datasets in Snowflake for extremely high-fidelity detections. Hunters.AI has shown us (Snowflake itself is a customer) that better data enables a new kind of security program. Its automated triage eliminates hundreds or thousands of daily alerts, giving security analysts more time to work strategically.
Why is Hunters.AI able to catch security breaches so much more effectively than other solutions on the market? Although machine learning is a necessary component of security detection (static rules don’t scale well), ML on its own is not enough. A single dimension of data, such as AWS CloudTrail, is never going to support accurate detections—no matter the algorithm.
Hunters.AI’s unique breakthrough technology enables its customers to intelligently connect information across multiple datasets. By combining logs from laptops, cloud infrastructure, and servers into one detection mechanism, Hunters.AI eliminates false positives while identifying attacker techniques that would otherwise go unnoticed. It’s an approach that has proven itself in multiple red team engagements at Snowflake and for other customers.
Snowflake’s partnership with Hunters.AI is also important because it overcomes the initial hurdle of data collection. Because Hunters.AI must serve customers that haven’t created a security data lake, it has data connectors for all the security data sources that traditional ETL vendors don’t cover. These include firewall logs, EDR, and cloud activity. With Hunters.AI, our customers don’t need to build their own integrations to start a security data platform.
The following diagram shows the data collection and threat detection integration:
Organizations facing the following issues should consider building a Snowflake security data platform with Hunters.AI:
- Security systems and budgets that can’t keep up with increasing log volumes
- Known visibility gaps and narrow retention windows
- Lack of detection coverage for the MITRE Cloud ATT&CK Matrix
Seeing our customers develop new use cases is one of the best parts of working at Snowflake. Our platform supports many types of data and applications, resulting in constant innovation. At the same time, some challenges are best solved by dedicated vendors that build repeatable solutions.
Hunters.AI’s solution for threat hunting on Snowflake is an example. To see our joint solution in action, visit us at the RSA conference this month. Click here to learn more about the offering and book a live demo.