Skip to content
  • AT SNOWFLAKE
  • Industry solutions
  • Partner & Customer Value
  • Product & Technology
  • Strategy & Insights
Languages
  • Deutsch
  • Français
  • Português
  • Español
  • English
  • Italiano
  • 日本語
  • 한국어
  • Deutsch
  • Français
  • Português
  • Español
  • English
  • Italiano
  • 日本語
  • 한국어
  • AT SNOWFLAKE
  • Industry solutions
  • Partner & Customer Value
  • Product & Technology
  • Strategy & Insights
  • Deutsch
  • Français
  • Português
  • Español
  • English
  • Italiano
  • 日本語
  • 한국어
  • 개요
    • Why Snowflake
    • 고객 사례
    • 파트너 네트워크
    • 서비스
  • 데이터 클라우드
    • 데이터 클라우드
    • 플랫폼 개요
    • SNOWFLAKE 데이터 마켓플레이스
    • Powered by Snowflake
    • 라이브 데모
  • WORKLOADS
    • 협업
    • 데이터 사이언스&머신러닝
    • 사이버 보안
    • 애플리케이션
    • 데이터 웨어하우스
    • 데이터 레이크
    • 데이터 엔지니어링
    • 유니스토어
  • PRICING
    • Pricing Options
  • 산업별 솔루션
    • 광고, 미디어 및 엔터테인먼트
    • 금융 서비스
    • 의료 및 생명 과학
    • 제조
    • 공공 부문
    • 소매 / CPG
    • 테크놀로지
  • 리소스
    • 리소스
    • Documentation
    • 핸즈온 랩
    • 트레이닝
  • CONNECT
    • Snowflake 블로그
    • 커뮤니티
    • 이벤트
    • 웨비나
    • 팟캐스트
  • 개요
    • 회사 소개
    • 투자정보
    • 리더십 및 이사회
    • 채용
Author
Dinesh Kulkarni Dinesh Kulkarni
Share
Subscribe
2020년 03월 19일 3 min read

Auto-ingest Snowpipe is Now Generally Available

  • 회사에 관한 정보
    • Snowflake 뉴스
  • 제품 및 기술
    • 데이터 웨어하우스
    • 데이터 엔지니어링
Auto-ingest Snowpipe is Now Generally Available

Real-time analytics are a key-enabler for making faster, data-driven decisions. Decisions made in haste with stale data or with no data can backfire, leaving organizations at a competitive disadvantage. Snowflake is in the business of enabling organizations to be data driven to capture the competitive advantage and data ingestion is a key piece of the puzzle.

In response to the increasing demand for real-time analytics, in late 2017 we introduced Snowpipe to help customers load all their data efficiently with a continuous, auto-scaled service with a simple REST API. The API took a list of files and could be called continuously. Then the following year we added an effortless auto-ingest option to eliminate the need for writing a driver that called the API periodically. Auto-ingest simply listens to the notifications from the cloud blob store such as Amazon S3 or Azure Blob Storage, and ingests new files as they arrive. We are excited to announce that the Auto-ingest Snowipe feature is now generally available on AWS and Azure in all regions. It layers on top of the solid foundation of Snowpipe which has been generally available in the form of a REST API since Fall 2018.

Effortless, continuous ingestion is now the norm

Additional sources of continuous data keep emerging in businesses. They create both an opportunity for the business and a challenge for the already stretched IT and data engineering organizations. Existing connectors for the new data sources often create files in blob store and leave the rest to the IT organizations to manage. With Auto-ingest Snowpipe, customers can simply set up a pipe with blob store notifications and leave the ongoing management of ingestion to Snowflake. Thereafter, the Auto-ingest Snowpipe infrastructure takes care of expeditiously processing the new file notifications and ingests the files with dynamically scaled Snowflake-managed warehouses. 

Data for the Masses

Even during preview, our customers loaded over 10 billion files covering over 17PB of data across AWS and Azure in just the last twelve months, and they did so by using Auto-ingest Snowpipe. It became the fastest growing ingestion option by far and it continues to grow rapidly. We appreciate your trust and are continuously working to enhance the simplicity, efficiency, and reliability of this popular continuous ingestion option.

“Auto-ingest with Snowpipe allows us to ingest log data in real time, from an S3 bucket and continuously to Snowflake with 15 million rows of data loaded every single day,” said Amaury Dumoulin, Senior Data Scientist at Qonto. “The best thing for our team is that we can “fire and forget” with Snowpipe with no maintenance, no management, and hassle-free. The performance gain and cost saving is across the board.” Raphaël Champeimont, Lead Architect at Doctrine France, also commented, noting that, “We leverage auto-ingest with Snowpipe for several of our data pipeline workloads. For example, we were able to automatically ingest data and capture changes from AWS Kinesis through Firehose to S3 and to our Snowflake table without setting up manual “COPY INTO,” and the data is available in almost real time. The cost performance is satisfying as we are able to cut down our pipeline cost by as much as 16 times.”  

Try Auto-ingest snowpipe today

Give Auto-ingest Snowpipe a spin today and let us know your feedback. You can find the documentation and information on how to get started here.

Share

Snowflake Snowpipe

Learn more about how to automate and continuously load data.

Learn More
Read More

Big Data as a Service

Big Data as a Service (BDaaS) integrates many of the functionalities and benefits of SaaS, IaaS, PaaS and DaaS, and leverages additional resources in the market for analyzing Big Data.

More to follow
Read More

Schema Detection Public Preview

Data pipelines are the lifeblood of modern analytics, which are a key enabler for making faster, data-driven decisions....

Full Details
Read More

The Price Is Right: Optimizing Retail Price Promotions with Big Dat...

In this article, we’ll explore how modern retailers use data-driven pricing and promotion strategies to increase sales and improve their bottom line.

Explore
Read More
Snowflake Inc.
  • 플랫폼 개요
    • 아키텍처
    • 데이터 애플리케이션
  • 데이터 마켓플레이스
  • Snowflake 파트너 네트워크
  • 지원 및 서비스
  • 회사
    • 문의하기

Sign up for Snowflake Communications

Thanks for signing up!

  • Privacy Notice
  • Site Terms
  • Cookie Settings

© 2023 Snowflake Inc. All Rights Reserved