Mercury Turns Fresh Data into Faster Product Decisions with Snowflake Openflow
With Snowflake Openflow, Mercury can trust the numbers behind its decisions, ship product updates faster and put engineering time back into building the business.
1-3Minutes of data latency to Snowflake, down from 30-60+ minutes
0Replication incidents since the migration
Industry
Financial ServicesLocation
San Francisco, CaliforniaTransforming financial workflows for over 300,000 companies
Launching or running a business isn’t for the faint of heart. Between bringing an idea to life and navigating red tape, entrepreneurs have plenty on their plates without wrangling their finances into shape.
That’s where Mercury comes in. Trusted by more than 300,000 customers, including one in three U.S. startups,* Mercury is a financial technology (fintech) company providing radically different banking.** It has changed how businesses manage money with a single account that replaces the patchwork of separate tools for banking, bill pay, invoicing, expenses, accounting and more.
Processing over $20 billion in monthly transactions is a data-intensive endeavor. Nearly every key decision at Mercury traces back to one operational database, and how fast and how completely that data reaches Snowflake sets the limit on what the company can know about its customers, its product and its risk.
With Snowflake Openflow, having access to timely data, but also correct data, helps us make our best decisions."
Nazanin Mirarab
Staff Engineer, Data Platform, Mercury
Story highlights
Giving every team a complete, trusted view of the business: Snowflake Openflow continuously loads Mercury’s full operational database, the same one that runs its app, into Snowflake without weighing it down, so teams across the company can trust the numbers they use to run the business.
Serving new business users with current data: Now that data in Snowflake is fresh, Mercury can serve business users who need timely answers, paving the way for use cases like fraud detection, faster product launch reporting, near real-time ML inference and new customer-facing features.
Slashing resync time from a week to days: A full database resync that used to take over a week and disrupt end users now runs in about two days with Openflow.
Outgrowing a data pipeline that couldn't keep up with the business
Mercury’s radically different approach to business banking is built on its customer-facing app that brings together checking, savings, credit cards, bill pay, invoicing, accounting and more. Running it depends on constant analysis of data from Mercury’s production database. “It basically drives all the decision-making about our business and helps us understand customers,” says Nazanin Mirarab, Staff Engineer, Data Platform at Mercury. “It’s a very critical data source.”
Previously, Mercury ingested data into Snowflake through a batch pipeline that struggled to keep pace with growth. Data that should have loaded in minutes took up to two hours or simply failed to appear, leading to blind spots for data scientists and analysts. Resynchronizing Mercury’s entire database took about a week, meaning customer dashboards went stale, analytics often turned unreliable and the team had to wait too long to act. “I don’t want our stakeholders to wait a week,” Mirarab says. “One or two days is OK, but a week is a problem.”
Having outgrown its prior solution, the team turned to Snowflake Openflow, Snowflake’s managed integration service built on Apache NiFi. “We needed something that could process at much higher volumes and scale while maintaining relative cost efficiency,” says Dan Goldberg, Data Engineering Manager at Mercury. “Openflow creates opportunities for end-to-end pipelines that exist entirely within Snowflake.”
Mercury’s engineers worked closely in a co-development cycle with Snowflake’s Openflow team, paving the way for a rapid three-month migration with zero disruption for end users. That collaboration included frequent feedback and iteration on production needs. One example of that collaboration came when Mirarab and Niko Klanecek, Staff Infrastructure Engineer at Mercury, proposed using a read replica for snapshot syncs, rather than putting additional load on Mercury’s production database. The Openflow team added support for the approach about a week later.
Building a 360-degree view of Mercury with Snowflake Openflow
With Openflow, Mercury now ingests its entire production database into Snowflake, not a subset, so nothing is missing when analysts and data scientists go looking. The data engineering team spends far less time debugging and more time building.
The clearest evidence is what the data engineering team no longer hears. “So far we really haven’t been having negative reports,” Mirarab says. “Data is there, it’s timely and available. Silence means things are good.” The runaway replication lag that once threatened the production database hasn’t recurred. “I don’t think we’ve had it happen once,” says Klanecek.
Openflow also gives Mercury observability that Mirarab’s feedback helped shape, the kind of visibility Mercury couldn’t get from managed extract, load and transform (ELT) vendors. Mercury’s engineers built an agent that calls Openflow directly to monitor syncs, so any engineer can check on a pipeline and troubleshoot it in a fully self-serve way. Now, the team can proactively detect and resolve sync issues on its own. “It’s been a very big unlock for us,” Mirarab says.
~2 Days to resync database, down from a week previously
~20 Hours per month of data engineering maintenance and firefighting removed
Maximizing ROI with less infrastructure and more control
For a company handling sensitive financial data, the usual paths fell short: An off-the-shelf connector meant giving up control, while a homegrown pipeline meant endless maintenance. Openflow pairs the control of a self-built system with the simplicity of managed infrastructure, and the overhead Mercury used to carry running its prior vendor’s system in-house is largely gone.
Goldberg weighed the full cost picture. “The numbers come out quite favorably on a per-unit basis when considering our prior bring-your-own-compute implementation, vendor and loading costs, engineering time and the fact that we couldn’t load all the data,” he says. “Now we get a lot more, and we’re in the same ballpark.”
Klanecek was initially a skeptic. Mercury consolidated most of its tooling into one cloud provider over the years, and he repeatedly argued for doing the same with its warehouse. He came around, seeing Snowflake as a platform rather than just a data store. Snowflake is now one of the very few third-party tools at Mercury to survive that consolidation, “just because of how much people love it,” Klanecek says.
With Snowflake Openflow, we're managing less infrastructure, but we have more control over it. It's a big win."
Niko Klanecek
Staff Infrastructure Engineer, Mercury
Making faster product decisions and building toward new customer features
Mercury’s data engineering team is working to provide a near real-time understanding of how customers use the Mercury app. “Openflow makes it possible for us to go from a product launch to version two, three and four much faster while maintaining a data-driven approach that’s fundamentally based in reality,” Goldberg says.
Snowflake Openflow opens new possibilities for Mercury’s data-savvy business users, whose demand for fresh data sometimes results in running their queries against a replica of the production database that was never built for analytics. “We’re unlocking a whole new class of users and use cases with fresh data in our warehouse,” Goldberg says. “It lets us move the querying off our production backend, deliver analytics use cases and build ML inference that is close to real time.”
There was no reason to build an app connector before, Klanecek says, because the data was too stale for anyone to want it. Now in pilot, Mercury’s newly developed Snowflake connector makes it possible to surface customer-facing data from Snowflake. According to Goldberg, “It creates whole new things that we can potentially do to serve our customers, like cash flow forecasting and other new value-add features.”
Compounding the value of one data platform
With complete data now landing continuously, Mercury is turning its attention to automating real-time operational workflows, from streamlining customer onboarding to accelerating service resolution and tightening risk controls, all powered by data that’s always current.
“Data is how we make our decisions and it’s the path to our success,” Goldberg says. “When it’s all in one place together in Snowflake, that’s when we start to see that accelerated value.”
With each new data innovation, Mercury continues to help hundreds of thousands of entrepreneurs truly understand their money as they dream and build what’s next.
*Calculation based on U.S.-based companies that received an angel, pre-seed, seed or Series A investment reported on Crunchbase in the most recent year.
**Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column N.A., Members FDIC.
