Snowflake customers want to discover insights from their data faster than ever, which is challenging because data volumes are growing at a breakneck pace. Effective searches are critical to customer satisfaction. Today, we’re excited to announce the general availability of search optimization, which significantly improves the performance of selective queries on large tables.
The search optimization service is an example of Snowflake’s focus on accelerating performance for a variety of workloads and enabling data engineers, data scientists, and BI analysts to work effectively with massive data sets within Snowflake.
Using The Search Optimization Service
The search optimization service enables users to analyze billions of rows of data, giving them faster insight on specific narrow questions. For example, data scientists can run several ad-hoc queries to make sense of a given data set, and data analysts can find useful insights that were previously hidden.
During the preview period, many Snowflake customers experienced much faster searches to find specific and small answers in massive data sets like transactions or activities for one user or how a single coupon code was used on a particular item. In some, specific cases, search times decreased from several minutes to a few seconds, as shown in the following table:
| Customer | Use Case | Query without Search Optimization Service | Query with Search Optimization Service | Improvement |
|---|---|---|---|---|
| Fortune 500 Financial Service Company | A lookup for financial transactions and account lookups for a given user | 5:16 minutes | 2.2 seconds | 143x |
| HYAS | A lookup of all the logs and data associated with one of many attributes | 37:14 minutes | 18.54 seconds | 120x |
| Fortune 500 Media Company | Lookups of specific user accounts and activity | 1:22 minutes | 16 seconds | 5x |
| Fortune 500 Retail Company | Looking for specific coupon usage on an item | 8.91 seconds | 2.3 seconds | 4x |
You can activate the search optimization service on selected tables and let Snowflake do the maintenance in the background. Snowflake automatically tracks additions, updates, or deletions while preserving the optimal access structure underneath.
In addition to speeding up your workloads, the search optimization service can cut costs by enabling you to use smaller virtual warehouses on existing workloads.
Randy Fox, Senior Software Engineer at HYAS, said, “Without search optimization, we would not be able to easily allow our customers to quickly interrogate hundreds of billions of rows of data without a big hardware footprint. Snowflake enabled us to serve our customers quickly and reliably with an end-to-end solution not mired in barely manageable archaic database paradigms. Layering search optimization on top has given us the speed and flexibility that we required, while keeping a simple data model.”
The feedback we received during the search optimization service preview highlights how important efficient data exploration is to all kinds of enterprises with critical data strategies and SLAs. Users described additional use cases involving more cost-effective security and log analytics. We will jointly explore these use cases further and provide updates in the coming months.
To learn more, see the detailed documentation and join us at our March 18 webinar.
Innovating as One Global Engineering Team
The general availability of the search optimization service marks a key milestone for Snowflake’s growing engineering teams in Europe. The search optimization service is the result of the work developed in Snowflake’s Berlin Development Center, in close collaboration with Snowflake’s engineering team in San Mateo, California. At the beginning of March 2021, the Berlin office consists of more than 30 engineers with diverse backgrounds, working together to enhance the Snowflake stack. If you’d like to learn more and help our European team build the future of Snowflake, see our careers page and select the Berlin location.