Intercontinental Exchange and the NYSE Optimize Costs With Snowpark

With Snowflake, Intercontinental Exchange has lowered costs and increased operational efficiency while supporting regulatory compliance and reporting for global exchanges like the New York Stock Exchange.



Better ad hoc query performance


Cost savings for NYSE’s regulatory reporting workload

Two financial advisors focusing on a report on a laptop, collaborating with each other
Financial Services
New York, NY

Investing in the right data platform

As a global financial organization with operations from the U.S. and Europe to Singapore and Abu Dhabi, Intercontinental Exchange operates exchanges, clearing houses, data services and mortgage technology. Part of ICE’s portfolio, The New York Stock Exchange (NYSE) is the world’s largest stock exchange, offering icons and entrepreneurs the opportunity to raise capital and change the world.

ICE’s strategy of connecting people to opportunity relies on innovative technology, industry expertise and massive amounts of data. As a result, both ICE and the NYSE turned to Snowflake’s Data Cloud to efficiently manage data at scale while also keeping costs under control.

Story Highlights
  • Better performance at scale: Switching to Snowflake provides ICE with near-infinite scalability to improve ad hoc reporting performance by up to 80% while handling billions of transactional records per day.

  • More than 50% cost savings: Within one month of deploying Snowpark for the NYSE’s regulatory reporting workload, related data costs fell by more than half.

  • Innovation and collaboration: With Snowflake, ICE is better positioned to experiment with emerging data technologies and enable advanced analytics.

Faster insights, greater scale, better visibility

ICE processes billions of transactional records every day — which places strain on its downstream analytical systems for regulatory compliance, market surveillance and customer reporting. “Between all of our systems, we’ve handled more than a half trillion messages on a single trading day,” says Anand Pradhan, Senior Director of Regulatory and NMS Tech at the New York Stock Exchange. “All these transactions happen in microsecond granularity, so that produces complex, dense time-series data.”

ICE’s previous on-premises data warehouse offered limited storage capacity, which led to time-consuming data archiving and costly hardware upgrades. To mitigate concurrency issues and reporting delays, technical staff spent time engineering workarounds instead of onboarding new workflows. 

Needing to meet increased demand for analytics, ICE began evaluating multiple cloud technologies. After in-depth proof of concept (POC) testing, ICE selected Snowflake’s Data Cloud. “We began our Snowflake journey in 2019 and did our POC with the derivatives exchange,” says ICE Director Divakar Jeyadevan. “We went live in 2020 during the peak of the pandemic, and it was very successful.”

Snowflake’s near-instant, near-infinite scaling of storage and compute supports ICE’s diverse reporting needs while handling terabytes of new data that’s loaded daily. Migrating workloads to the Data Cloud has improved ad hoc query performance by up to 80% and practically eliminated service-level agreement (SLA) issues for data availability.

With Snowflake, we are getting nearly infinite, low-cost storage that is cloud-agnostic. The separation of storage from compute results in excellent concurrency and scalability.”

Durgesh Das
Head of Data Warehousing, Analytics & Governance, Intercontinental Exchange

ICE now enjoys a 360-degree view of business and operational metrics, including enhanced cost transparency. “With so many business units, Snowflake provides a good view of cost management to simplify administration,” Jeyadevan says. Snowflake’s fully managed infrastructure also allows ICE’s technical staff to focus on gaining analytics — not procuring hardware, performing system upgrades or juggling workarounds. 

Cutting costs by over 50% from NYSE’s migrated Snowpark workloads

While migrating existing workloads to the Data Cloud was a critical first step in ICE’s Snowflake journey, the focus quickly shifted to maximizing on-cloud savings, increasing throughput and accelerating data’s impact. 

Optimizing the NYSE’s mission-critical regulatory reporting workload presented a big opportunity to reduce data costs and increase productivity. “Our regulatory reporting workload looks at hundreds of billions of records in the data set and constantly merges them together to find different patterns — all while joining various reference data sources. It requires a massive amount of compute,” says Pradhan. 

The NYSE originally used Snowflake with a Spark connector to process data engineering pipelines in an external environment. Yet maintaining two separate compute environments was costly and operationally burdensome, yielding both hidden costs and “double taxation” from moving data in and out of Snowflake. 

As a result, the NYSE turned to Snowpark for a cost-efficient, reliable way to power its regulatory reporting workload. Collaborating with Snowflake Professional Services expedited the Snowpark transition while reducing risk. “We migrated to Snowpark without any governance trade-offs since Snowpark allows us to build Python-based data engineering pipelines all in a single Snowflake platform, where our data already lives,” says Pradhan. “It has been a significant step and massive success for us.”

Intercontinental Exchange quote card image

“We picked the most difficult workloads to engage with Snowflake Professional Services on. They helped us design and increase performance across pipelines and business functions, and their collaboration allowed us to prove our success with Snowpark and move to production in a very short period of time.”

Anand Pradhan
Senior Director, Regulatory Technology, NYSE

With Snowpark, the NYSE eliminates complexity and saves money. Related data costs fell by more than 50% within one month of deploying Snowpark for its regulatory reporting workload — despite relatively consistent data volumes. According to Pradhan, “Cost reduction with Snowpark was so impactful that I had to adjust my budget for this year.” 

Streamlining workloads with Snowpark also led to fewer points of failure and less manual code tuning to manage. “Because we have fewer operational challenges, our data engineering throughput increased by manyfold,” says Pradhan. “We now focus more on advanced analytics and better utilize our time to improve productivity.”

With any other solution, it’s very difficult to get cost transparency. With Snowpark, everything happens within Snowflake and you can see the exact costs.”

Durgesh Das
Head of Data Warehousing, Analytics & Governance, Intercontinental Exchange

Rapidly adapting to support users’ evolving analytics needs

Connecting SaaS tools to Snowflake makes it easier for ICE to maximize data for better business outcomes. “After our initial successes, our developer community is excited to take advantage of Snowpark for Python and exploring more use cases to migrate from Spark,” says Jeyadevan. Pradhan’s team is currently evaluating solutions to support machine learning and generative AI use cases, as well as testing some of Snowflake’s latest AI/ML features. According to Pradhan, “All these opportunities opened up for us because we started partnering with Snowflake, and we hope that continues to grow.”

Optimizing additional workloads will put ICE in an even better position to respond to changing market conditions and rising data volumes. Snowflake Marketplace will likely play an important role in future optimizations. Pradhan says, “Migrating to Snowflake was the right step forward so that we can do these different experiments, innovations and collaborations.”

Start your 30-DayFree Trial

Try Snowflake free for 30 days and experience the AI Data Cloud that helps eliminate the complexity, cost and constraints inherent with other solutions.