Real-time analytics are a key-enabler for making faster, data-driven decisions. Decisions made in haste with stale data or with no data can backfire, leaving organizations at a competitive disadvantage. Snowflake is in the business of enabling organizations to be data driven to capture the competitive advantage and data ingestion is a key piece of the puzzle.
In response to the increasing demand for real-time analytics, in late 2017 we introduced Snowpipe to help customers load all their data efficiently with a continuous, auto-scaled service with a simple REST API. The API took a list of files and could be called continuously. Then the following year we added an effortless auto-ingest option to eliminate the need for writing a driver that called the API periodically. Auto-ingest simply listens to the notifications from the cloud blob store such as Amazon S3 or Azure Blob Storage, and ingests new files as they arrive. We are excited to announce that the Auto-ingest Snowipe feature is now generally available on AWS and Azure in all regions. It layers on top of the solid foundation of Snowpipe which has been generally available in the form of a REST API since Fall 2018.
Effortless, continuous ingestion is now the norm
Additional sources of continuous data keep emerging in businesses. They create both an opportunity for the business and a challenge for the already stretched IT and data engineering organizations. Existing connectors for the new data sources often create files in blob store and leave the rest to the IT organizations to manage. With Auto-ingest Snowpipe, customers can simply set up a pipe with blob store notifications and leave the ongoing management of ingestion to Snowflake. Thereafter, the Auto-ingest Snowpipe infrastructure takes care of expeditiously processing the new file notifications and ingests the files with dynamically scaled Snowflake-managed warehouses.
Data for the Masses
Even during preview, our customers loaded over 10 billion files covering over 17PB of data across AWS and Azure in just the last twelve months, and they did so by using Auto-ingest Snowpipe. It became the fastest growing ingestion option by far and it continues to grow rapidly. We appreciate your trust and are continuously working to enhance the simplicity, efficiency, and reliability of this popular continuous ingestion option.
“Auto-ingest with Snowpipe allows us to ingest log data in real time, from an S3 bucket and continuously to Snowflake with 15 million rows of data loaded every single day,” said Amaury Dumoulin, Senior Data Scientist at Qonto. “The best thing for our team is that we can “fire and forget” with Snowpipe with no maintenance, no management, and hassle-free. The performance gain and cost saving is across the board.” Raphaël Champeimont, Lead Architect at Doctrine France, also commented, noting that, “We leverage auto-ingest with Snowpipe for several of our data pipeline workloads. For example, we were able to automatically ingest data and capture changes from AWS Kinesis through Firehose to S3 and to our Snowflake table without setting up manual “COPY INTO,” and the data is available in almost real time. The cost performance is satisfying as we are able to cut down our pipeline cost by as much as 16 times.”
Try Auto-ingest snowpipe today
Give Auto-ingest Snowpipe a spin today and let us know your feedback. You can find the documentation and information on how to get started here.