Many organizations use Snowflake for security analytics, regulatory compliance, and anti-fraud. This includes using Snowflake as a Security Information and Event Management (SIEM) product. They put massive amounts of relevant data into Snowflake and use its scalable architecture to quickly detect, investigate, and report on cyberthreats and fraudsters.
One such organization includes Snowflake, as our internal Information Security (InfoSec) team, which I manage, is all-in on using Snowflake for security analytics. In this blog post we’ll share the details of our journey from another security analytics/SIEM product to Snowflake.
THE BACKGROUND
Snowflake stores critical business data for its customers and protecting this data is paramount. We need to be able to detect any threat action or activity such as compromised accounts, lateral movement, and data exfiltration.
To help with this, years ago we implemented a cloud-based SIEM solution to index events and log data from Snowflake’s internal security and IT products and harness this data to detect and defeat cyberthreats. We evaluated only cloud-based solutions because of the advantages they had over on-premises solutions, including faster time to value, no costly physical appliances, lower maintenance costs, and more.
CHALLENGES WITH OTHER CLOUD SECURITY ANALYTICS SOLUTIONS
Snowflake’s InfoSec team very quickly started encountering challenges with this cloud-based product including:
- A high percentage of false positives. It had pre-built rules that were not tailored for Snowflake’s unique environment, and it was difficult to create custom, accurate rules in it. The result was daily alerts often exceeding 2000, a number which was not manageable and led to both alert fatigue and concerns a valid threat could slip by.
- Scalability issues. A weak architecture meant scaling up to handle more data and complex searches was difficult, and it led to extremely slow searches.
- Limited extensibility and flexibility. For certain data sources, especially custom applications, it was very difficult to index their logs. It was also difficult to customize the UI to visualize and investigate threats in the way we wanted to.
- High costs. License costs were so exorbitantly high that we were not able to index all the data sources we wanted to, nor were we able to store data for the long time periods desired. Log retention was measured in days and we were concerned that if we needed to investigate a serious issue, the incident response data set would be full of holes.
We knew something better was needed for security analytics. It was time for a new approach.
SNOWFLAKE FOR SECURITY ANALYTICS: DEPLOYMENT AND ARCHITECTURE
At that time, we knew at least one Snowflake customer, a Fortune 100 financial services company, was using Snowflake for better, automated security analytics versus using traditional SIEM, and that’s when we realized the answer was right in front of us. Snowflake! Sure, data warehouses are not usually associated with security analytics, but that was before Snowflake and its unique architecture came along. Snowflake is a cloud-based, highly scalable, flexible, cost-effective platform that can ingest and store all data, including semi-structured data, in one place to accomplish all key SIEM use cases: queries/detections, investigations, reporting, and compliance. So we went about building out Snowflake for security analytics over the course of a few weeks.
Our cloud activity logs were already collected in cloud buckets as JSON so it was easy for Snowflake’s automated Snowpipe service to ingest them. Logs from on-premises servers and applications were collected in cloud buckets via open source agents such as osquery, Fluentd and Elastic Beats. For SaaS applications, we polled the applications’ REST API endpoints with AWS Lambda functions. We also put contextual, “non-event” data into Snowflake. This data included asset inventory, cloud configuration, and user directory data. This contextual information was weaved into detection rules for better accuracy.
To make it easier for Snowflake customers’ InfoSec teams to do the same, my team bundled much of this implementation work into a free, open-source analytics framework called SnowAlert. Within SnowAlert, detection rules are defined in highly accurate SQL, run on a schedule, and results are stored back in Snowflake for investigation and reporting. We also included JIRA integration and SQL query packs with pre-built content for common use cases. SnowAlert is open to contributions from all users; it’s a great place for collaboration between security engineers. Courtesy of our partner Hashmap, read more on SnowAlert here.
Lastly, we used a cloud-based, intuitive business intelligence product as our user interface to view, investigate, and report on alerts, and to view our overall security posture. Over the first few weeks of deployment, we fine-tuned our SQL queries/detection rules and refined our user interface incident review workflow and reports. We ended up with a complete security analytics solution perfectly tailored for our unique environment and processes.
BENEFITS OF SNOWFLAKE FOR SECURITY ANALYTICS
When we were done, the benefits were immediate and substantial, and they have improved with time. Benefits include:
- High accuracy. The flexible and powerful SQL language lets my team write queries tailored for our unique environment and the threats we face. The result is high accuracy and minimal false positives. There is no longer alert fatigue because the typical daily alert volume has been reduced to under 20 tickets. My team can now investigate every detection to its root cause, and suspicious activity that before would have been lost in the noise is now closely analyzed. The team also has more time to do threat modeling and write more detection rules for new use cases.
- Unlimited, elastic scalability. Snowflake’s unique cloud-built architecture separates storage and compute and can auto-scale up and out with essentially no limit. A single workload, whether it’s the ingestion of event data or a SQL query, can get the dedicated compute resources it needs to be blazing fast. My team now sees extremely fast queries that drive quick threat detection and investigations, and there is no need to limit concurrent searches or users.
- Extensibility and flexibility. Snowflake’s extensibility makes it easy to get events and logs from any data source, even custom apps, into Snowflake. It can also accommodate very custom, tailored SQL queries and work with essentially any business intelligence or advanced analytics/data science product. By using a business intelligence product with an easy-to-use UI, Snowflake’s InfoSec team can quickly, and easily, build any custom interface or report we need.
- Reduced costs. Snowflake ended up costing over 90% less than the prior solution. Much of this comes from Snowflake’s fair pay-as-you-use model, which charges only for compute resources as they are used. Some savings are from Snowflake passing through the cloud provider’s very low storage costs of $23/TB/month after compression; this low storage cost actually now lets us store all our event data indefinitely with no need to delete any of it. Lastly, lowered costs also come from Snowflake’s ease of use and near-zero maintenance.
The solution has worked so well that other internal teams have gotten involved. The Snowflake Compliance team indexes data in Snowflake and builds reports and dashboards they need to prove compliance with technical controls to auditors. The Snowflake IT and Operations team view operational metrics relevant to them in the data.
SCREENSHOTS
Because a picture is worth a thousand words, here are some screenshots of what the internal Snowflake for Security Analytics deployment looks like:
CLOSING
So that was the journey of Snowflake’s internal InfoSec team from knowing we needed an analytics solution, to struggling with a traditional solution, to succeeding with Snowflake. The net result has been much better, faster, and lower-cost detection, investigations, reporting, and compliance… all driven by rich data.
If you are interested in learning more, see the Snowflake Security Analytics web page and read the solutions guide. To speak with Snowflake Sales to learn more, see a demo, or collaborate with us on SnowAlert, go to this web page.
Thanks for your interest in Snowflake and letting us share our journey with you.
Omer Singer, Senior Director of Security
Mario Duarte, VP of Security
Joe Goldberg, Director of Product Marketing