Intercontinental Exchange and the NYSE Optimize Costs With Snowpark

As a global financial organization with operations from the U.S. and Europe to Singapore and Abu Dhabi, Intercontinental Exchange operates exchanges, clearing houses, data services and mortgage technology. Part of ICE’s portfolio, The New York Stock Exchange (NYSE) is the world’s largest stock exchange, offering icons and entrepreneurs the opportunity to raise capital and change the world.

ICE’s strategy of connecting people to opportunity relies on innovative technology, industry expertise and massive amounts of data. As a result, both ICE and the NYSE turned to Snowflake’s Data Cloud to efficiently manage data at scale while also keeping costs under control.

ICE processes billions of transactional records every day — which places strain on its downstream analytical systems for regulatory compliance, market surveillance and customer reporting. “Between all of our systems, we’ve handled more than a half trillion messages on a single trading day,” says Anand Pradhan, Senior Director of Regulatory and NMS Tech at the New York Stock Exchange. “All these transactions happen in microsecond granularity, so that produces complex, dense time-series data.”

ICE’s previous on-premises data warehouse offered limited storage capacity, which led to time-consuming data archiving and costly hardware upgrades. To mitigate concurrency issues and reporting delays, technical staff spent time engineering workarounds instead of onboarding new workflows. 

Needing to meet increased demand for analytics, ICE began evaluating multiple cloud technologies. After in-depth proof of concept (POC) testing, ICE selected Snowflake’s Data Cloud. “We began our Snowflake journey in 2019 and did our POC with the derivatives exchange,” says ICE Director Divakar Jeyadevan. “We went live in 2020 during the peak of the pandemic, and it was very successful.”

Snowflake’s near-instant, near-infinite scaling of storage and compute supports ICE’s diverse reporting needs while handling terabytes of new data that’s loaded daily. Migrating workloads to the Data Cloud has improved ad hoc query performance by up to 80% and practically eliminated service-level agreement (SLA) issues for data availability.

ICE now enjoys a 360-degree view of business and operational metrics, including enhanced cost transparency. “With so many business units, Snowflake provides a good view of cost management to simplify administration,” Jeyadevan says. Snowflake’s fully managed infrastructure also allows ICE’s technical staff to focus on gaining analytics — not procuring hardware, performing system upgrades or juggling workarounds. 

While migrating existing workloads to the Data Cloud was a critical first step in ICE’s Snowflake journey, the focus quickly shifted to maximizing on-cloud savings, increasing throughput and accelerating data’s impact. 

Optimizing the NYSE’s mission-critical regulatory reporting workload presented a big opportunity to reduce data costs and increase productivity. “Our regulatory reporting workload looks at hundreds of billions of records in the data set and constantly merges them together to find different patterns — all while joining various reference data sources. It requires a massive amount of compute,” says Pradhan. 

The NYSE originally used Snowflake with a Spark connector to process data engineering pipelines in an external environment. Yet maintaining two separate compute environments was costly and operationally burdensome, yielding both hidden costs and “double taxation” from moving data in and out of Snowflake. 

As a result, the NYSE turned to Snowpark for a cost-efficient, reliable way to power its regulatory reporting workload. Collaborating with Snowflake Professional Services expedited the Snowpark transition while reducing risk. “We migrated to Snowpark without any governance trade-offs since Snowpark allows us to build Python-based data engineering pipelines all in a single Snowflake platform, where our data already lives,” says Pradhan. “It has been a significant step and massive success for us.”

With Snowpark, the NYSE eliminates complexity and saves money. Related data costs fell by more than 50% within one month of deploying Snowpark for its regulatory reporting workload — despite relatively consistent data volumes. According to Pradhan, “Cost reduction with Snowpark was so impactful that I had to adjust my budget for this year.” 

Streamlining workloads with Snowpark also led to fewer points of failure and less manual code tuning to manage. “Because we have fewer operational challenges, our data engineering throughput increased by manyfold,” says Pradhan. “We now focus more on advanced analytics and better utilize our time to improve productivity.”

Connecting SaaS tools to Snowflake makes it easier for ICE to maximize data for better business outcomes. “After our initial successes, our developer community is excited to take advantage of Snowpark for Python and exploring more use cases to migrate from Spark,” says Jeyadevan. Pradhan’s team is currently evaluating solutions to support machine learning and generative AI use cases, as well as testing some of Snowflake’s latest AI/ML features. According to Pradhan, “All these opportunities opened up for us because we started partnering with Snowflake, and we hope that continues to grow.”

Optimizing additional workloads will put ICE in an even better position to respond to changing market conditions and rising data volumes. Snowflake Marketplace will likely play an important role in future optimizations. Pradhan says, “Migrating to Snowflake was the right step forward so that we can do these different experiments, innovations and collaborations.”