An Architect's Guide to Multi-Engine Lakehouses: What's Solved and What Isn't

Everyone in the data industry has heard the lakehouse pitch by now. Store your data once in an open format; attach any engine to it, depending on the job; swap compute in and out as requirements evolve; and retire the endless cycle of ETL between siloed systems. The benefits, when this is working, are concrete: no data duplication or synchronization headaches; the freedom to use specialized engines for different workloads rather than forcing everything through one tool; governance over a single copy of data, rather than auditing access sprawled across multiple systems; and meaningful insurance against the future. When your data is in an open format, new engines, vendor acquisitions, pricing changes and better tools for specific workloads all become compute decisions, not data migrations. It's a compelling vision. We completely agree with it.

And the good news is that the industry has made real, meaningful progress toward it. For the first time, we can genuinely see that the era of siloed systems — where every new tool means a new pipeline and another copy of the data to maintain — is approaching its end. The promise of storing data once and having multiple engines read it is exactly why organizations are investing so seriously in lakehouse architectures right now. Thanks to Apache Iceberg™ and the broader open table format ecosystem, true data portability is possible. That's not a small thing. As Figure 2 shows, the shift from fully siloed platforms, with their own separate data stores, to a world where data is shared and engines sit on top of a common foundation significantly reduces the pains of data architectures that are based on copying data all the time.

Apache Iceberg has been a driving force in making this possible. Because of its open specification, vendor-neutral project governance and broad cross-vendor adoption, it has become the de facto industry standard. Even Databricks, the creator of Delta Lake, has embraced Iceberg into its roadmap, spending up to a reported nearly $2 billion to acquire Tabular, a company founded by Iceberg's original creators. That move should only underscore the importance of Iceberg to the future of data lakehouses.

The real beauty of Iceberg, however, is that it is governed by the Apache Software Foundation (ASF) and not by any single company. With contributors from various organizations, including Bloomberg, Databricks and Snowflake, the Project Management Committee (PMC) oversees the development of Iceberg, with all members having equal votes (and veto rights) to decide what code gets merged. This helps ensure that the project is controlled fairly, since the PMC is held accountable to the ASF board. This also largely explains why Iceberg has seen such widespread adoption across the industry. As a community-driven project, it includes a governance structure that helps ensure that development benefits the ecosystem as a whole rather than one company's interests; it gets us closer to that aforementioned ideal state. As the old saying goes: If you want to go fast, go alone; if you want to go far, go together.

And yet, despite all the progress, the ecosystem has not yet converged on ways to satisfy all the production requirements for full interoperability¹. The challenge is that full interoperability hasn't been fully solved, decisions with trade-offs have to be made, which are neither easily identified nor well-documented. Organizations typically only identify gaps during implementation or after they've made significant architectural commitments that aren't easy to unwind. That's why when vendors say that interoperability is fully solved, it's important to question and validate their claims.

At Snowflake, we are committed to helping the multi-engine lakehouse become a reality for our customers. As we work with Apache Iceberg for data interoperability, Apache Polaris for governance interoperability and Apache Ossie (incubating) for semantic interoperability to work toward this vision, we want to help teams understand what is solved, what isn't, and how to avoid painful mistakes as they embark on this journey.

The problems we cover in this blog series aren't inherently lakehouse problems, or even multi-engine lakehouse problems. Rather, they are simultaneous multi-engine access problems. If you remove that requirement for a given data set, most of these problems go away. Whether to allow simultaneous multi-engine access in your architecture isn't a binary choice; it can be a per-workload or per-data set cost-benefit analysis, and the answer will be different for different parts of your architecture. It will also change over time as these problems get solved. For example, for workloads or data sets where the cons of simultaneous multi-engine access outweigh the pros, you can store everything in Iceberg format but restrict access to a single engine — whether that is Snowflake or any other that meets your requirements. Then, once the community solves the problems that led you to determine the cons outweigh the pros for that workload or data set, no migration will be needed. Your data is already in Iceberg format, so you can just open up additional engines to access it. We'll cover this in more depth later in this blog (see: The pragmatic path forward).

This post, along with three more deep-dive examinations in this series, is meant to highlight the gaps in the ecosystem today and then identifies specific strategic architectural decisions that data teams need to make in order to power production-ready, multi-engine environments. Our goal is to help organizations like yours avoid the pitfalls of discovering these issues too late and provide you with a practical guide to interoperability. In each section, we assess the current maturity of interoperability and any problem areas that the ecosystem hasn't fully solved, provide helpful guidance, and introduce questions to pressure test your architectural decisions.

The specific claims in this series are tied to publicly verifiable evidence — community mailing lists, open spec PRs, documented engine behaviors — and we'd encourage you to challenge any of it, as long as you take into account the state of the ecosystem at the time of writing and the velocity of change happening daily. Our aim is to distill these discussions into an actionable and pragmatic guide you can use to make architectural decisions.

A multi-engine lakehouse is an architecture that stores data once, in an open format, and lets multiple compute engines read and write to that data directly. It is effectively unbundling a database or data platform. For most of their 50-year history, databases have been a self-contained bundle: table storage, the query engine, transactions, governance and everything else, fused into a single, closed unit you bought and operated as one thing. A lakehouse pulls that unit apart, starting with the storage layer. But a database is much more than table storage. It bundles in numerous other capabilities that the industry has learned are essential in the history of using relational databases, such as ACID transactions, fine-grained access control and governance, query optimization, indexing and metadata management.

In this blog series, we organize the capabilities of data platforms into three categories: data interoperability, business logic interoperability and governance interoperability. They all need to be fully interoperable before the lakehouse vision is truly realized. But each has shortcomings that need to be addressed.

Of the three dimensions, data interoperability is in the best shape and is mostly solved. Apache Iceberg, paired with the Iceberg REST Catalog (IRC) as the access layer, has largely addressed the structural problem of sharing data across engines. Most major engines can read and write to Iceberg tables today, and the REST Catalog spec provides a vendor-neutral way to manage table metadata. This is genuinely broad: Spark, Trino, Dremio, Flink and others have mature Iceberg read and write support today, and the REST Catalog spec gives them a vendor-neutral way to manage table metadata. That's real and meaningful progress and is the community's achievement.

But, again, it is "mostly solved," and it's worth understanding what isn't solved yet. Iceberg v3 introduced important capabilities — native semi-structured type support, improved deletion tracking, row lineage — but v3 isn't an all-or-nothing proposition. When a vendor says it "supports v3," you need to ask which specific features it supports and verify those against your actual requirements. This will also likely be the case with v4. Also, upgrading a table from v2 to v3 is a one-way door, specified in the function upgradeFormatVersion. You can't do it until every engine interacting with that table can handle the baseline v3 format, and v3 support across the ecosystem is still uneven.

There's also a subtler, frequently overlooked problem: performance consistency across engines. Iceberg is intentionally unopinionated about file hygiene. That means without central enforcement, different write pipelines — especially in decentralized organizations — will produce inconsistent file sizes, row group sizes and column statistics configurations. What's optimal for one engine can be suboptimal for another, and the debugging can be nontrivial because degradation tends to show up gradually and at scale, or when someone suddenly changes a setting on the writing engine.

A concrete example: The Iceberg SDK's write.metadata.metrics.max-inferred-column-defaults setting defaults to 100, meaning full column statistics are only written for the first 100 columns for engines that don't override this setting. Spark uses this default, since it extends a class in the Iceberg Java library implementation that uses this default, and Spark doesn't apply any overriding of this value out-of-the-box. A query that changes to now filter on column 101 instead of column 100 can degrade badly, and the problem is often hard to catch until it's already affecting users. This is addressable regardless of engine: While the Iceberg library has this as a default, your write jobs on many engines can change this default when writing. The trade-off is that this is opt-in and operational, so someone is responsible for setting this configuration properly. Snowflake's Iceberg writer takes an opinionated view and defaults to writing column statistics for all columns. It also provides an AUTO file size setting, which can be overridden at the database, schema or table level if you need to tune based on what you know. This opinionated stance works well for the most common read-heavy patterns, though it trades off write throughput and metadata size in exchange.

Disaster recovery and cross-region access add another wrinkle. Standard object storage replication breaks the usability of Iceberg tables because file references in Iceberg metadata use absolute paths. According to the vote thread on the dev mailing list, the v4 spec will include a feature to support relative paths, which would eliminate this problem entirely by storing paths relative to the table's base location (though you'll still have to handle incrementalization for data replication). However, for now, production-ready, vendor-agnostic solutions aren't there yet for most architectures. Snowflake provides a replication and failover solution that addresses these issues for Snowflake-managed Iceberg tables.

Separately, and worth calling out explicitly: All engines reading from Iceberg tables should ideally be co-located in the same cloud region as the storage. Cross-region reads incur egress costs, degrade query performance, increase compute costs and expand your security surface in ways that are easy to underestimate at architecture time.

Key questions to pressure-test your data interoperability:

1. Are all your engines, platforms and catalogs IRC-compliant bidirectionally — that is, not just inbound or outbound, but in both directions, as well as for read vs. read and write?
2. Which specific Iceberg v2 and v3 features does each engine in your architecture actually support, and have you mapped that against your requirements, rather than assuming the "supports v3" marketing claim meets your needs?
3. How are you handling disaster recovery and cross-region access for tables that need it, given that standard object storage replication breaks Iceberg's absolute path references?

This is a top-level view of the data interoperability terrain. The deep dive covers the full Iceberg v3 feature support breakdown across engines; specific BC/DR approaches and their trade-offs; ML workload considerations, including wide tables and vector formats; and write pipeline configuration guidance.

Read our full deep dive on data interoperability →

This is where teams most reliably get caught off-guard. Shared storage is only part of what you're unbundling when you move to a lakehouse. Another part is business logic (that is, views, UDFs, stored procedures, metric definitions). And here, there is no complete, widely adopted standard. That is not for a lack of attempts but because existing efforts, such as Iceberg Views, Substrait and Ossie, haven't yet achieved the technical comprehensiveness and cross-engine adoption needed to be reliable architectural dependencies. For anyone making production architecture decisions today, the practical effect is the same as having no standard. However, we recommend understanding these projects and their trajectories so you don't make architectural decisions that box you out of using these projects once their business logic interoperability capabilities mature.

What this means in practice is that any metric or transformation defined in one engine must be manually redefined in every other engine accessing that data. There's no synchronization mechanism and no feedback loop that tells you when those definitions have drifted. The CFO gets different revenue numbers from two dashboards pointed at the same Iceberg table because the definitions quietly diverged months ago, and nobody caught it. This isn't theoretical; it's actually happening now. Companies like Airbnb and LinkedIn have already encountered this problem at scale, and they have built entire internal frameworks to address it — Minerva at Airbnb and Coral at LinkedIn (discussed later in this section) being two well-documented examples. You don't build and operate a whole framework to solve a theoretical problem. But those organizations have an army of engineers who can build and, critically, maintain these systems indefinitely. For most organizations this is a build-versus-buy decision that skews heavily toward buy; the ongoing maintenance burden alone tends to outweigh the initial build cost.

Iceberg Views exist and represent a step in the right direction, but the current implementation is essentially a metadata container: It basically stores the SQL text and the dialect it was written in (a few other properties too, but the main components are those fields). There's no translation layer between dialects, and behavioral variance between SQL engines is real. Null ordering, type casting, tie-breaking in window functions all differ across engines in ways that can produce subtly wrong results that are hard to catch without deliberate cross-engine regression testing. This is not impossible to address, but it's not automatic and definitely not out of the box.

Ossie, formerly Open Semantic Interchange which had initial contributions from organizations such as Snowflake, Salesforce, dbt Labs and others, was born from a real problem: Every tool in the stack was interpreting business metrics differently, eroding trust in AI-driven insights before they could get off the ground. Donated to the Apache Software Foundation (ASF) and now undergoing incubation, Ossie is a meaningful step for metrics and semantic definitions, though it's worth noting that it still doesn't address the broader scope of business logic objects, such as UDFs, stored procedures and procedural logic, which remain outside its purview. Transformation tools like dbt and SQLMesh can generate per-engine SQL from a higher-level spec, and they're worth exploring. The longer-term solution to the broader problem is likely something like an interoperable intermediate representation (IR) — a single, engine-agnostic representation of logic that gets adapted to each engine's dialect, similar to what LinkedIn built internally with Coral or what the Substrait project is working toward.

There's also an accountability gap that doesn't get discussed enough. In a single-engine world, there's one vendor to call when something produces the wrong answer. In a multi-engine world, debugging "why do these two engines produce different results from the same table" is a cross-vendor support problem — slower to resolve, ambiguous in ownership and costly when it surfaces in a downstream business decision.

Note that, while this is the least mature dimension of interoperability, there is a pretty good workaround available that addresses most of the problems: physicalize the business logic as part of a pipeline. That pipeline would compute the result of the business logic and write the results of that computation to a physical table so all engines inherently get the same answer. This workaround isn't free from trade-offs, however. We cover the possible workarounds and trade-offs in more depth in the business logic interoperability deep dive blog.

Key questions to pressure-test your business logic interoperability:

1. What's your plan for assessing current business logic that hasn't been physicalized into a table and, for that logic, modifying your environment to now physicalize it so multiple engines are guaranteed to have the exact same logic applied?
2. For any business logic you can't physicalize into a table — active users, recognized revenue, churn — how do you detect when definitions diverge across engines, and what's your reconciliation process?
3. When two engines produce different results from the same Iceberg table and neither vendor can reproduce the issue independently, what is your escalation path, and have you mapped that cross-vendor accountability before you're in production?

The section above captures the core problem. The deep dive goes further — covering the Iceberg Views and UDF specs in detail, Ossie in depth, transpilation approaches and their trade-offs, the potential use of dbt and SQLMesh to mitigate gaps, and concrete guidance on which architecture patterns hold up for which use cases.

Read our full deep dive on business logic interoperability →

Object-level access control is largely addressed. The catalog-vended, temporary scoped credential pattern — where the catalog holds long-lived storage credentials and issues short-lived tokens to engines when access is allowed — works well and is broadly supported. Most engines and platforms support this, but unfortunately, not all.

Fine-grained access control (FGAC), on the other hand, is a different story. Row- and column-level security is not fully standardized across platforms yet. If you define a row filter or column mask in one engine's catalog, there's no standard mechanism that guarantees another engine will apply the same policy when it reads that table that works well in all situations. Teams end up duplicating policies across platforms, which increases maintenance overhead and, more importantly, creates opportunities to introduce enforcement gaps — the kind that only surface during a compliance audit.

Snowflake is actively working with the communities to solve this by helping lead ongoing discussions in both the Iceberg and Apache Polaris projects around things like standardizing how to pass around FGAC rules as well as using the scan plan API in the IRC spec as a near-term mitigation. These are viable paths forward, but they each have pros and cons. Where one approach may make more sense for a given situation, a different approach could make more sense for a different situation (for example, small data set with infrequent changes with simple FGAC policies vs. large data set with frequent changes with complex FGAC policies). Furthermore, they're not yet production-ready in a consistent, vendor-agnostic way. There are additional approaches, such as defining them in a single system and having that system translate those policies and push them to each engine, that each have pros and cons.

Catalog federation complicates things further. Most multi-vendor deployments today rely on a cache-and-async-refresh model where a local catalog treats itself as the source of truth and periodically syncs from the remote. The consequence: Policies defined in your source catalog can be bypassed by a federated catalog operating from a stale state. There are, however, ways to narrow this window, including just-in-time refresh, which Snowflake recently implemented. But fresh metadata only closes the staleness and revocation-lag gap; it doesn't get a federated engine to enforce a policy it was never given. That harder work is splitting into two efforts in the Iceberg and Polaris communities, with contributors from multiple vendors: read restrictions, the nearer-term piece that lets any IRC-speaking engine enforce the row filters and column masks defined in your catalog (Iceberg PR #13879), and policy exchange, which sequences behind read restrictions to sync the policy definitions between catalogs so even engines bound to their own catalog enforce them (Polaris PR #2048).

Lineage is another casualty — fragmented across catalogs, with no standard mechanism to produce an end-to-end picture of data flow across engine boundaries. There is progress being made on this front with OpenLineage emerging as the de facto standard; however, it's generally still up to customers today to wire it all together. Snowflake's Horizon Context automates this work for you, ingesting lineage events from your broader data estate.

Data classification tags are another area where standardization remains incomplete. While there are emerging proposals in the open source community around basic label interoperability across catalogs, these are early-stage and don't yet address the richer tag semantics that most governance frameworks require — specifically, inheritance hierarchies that propagate classifications downstream through derived and dependent objects. There is currently no standard protocol for propagating full tag definitions and their inheritance behavior across catalog federation boundaries.

Transactional consistency across catalogs is difficult to guarantee. Atomic commit across two independently governed catalogs with no shared transaction coordinator is a fundamental distributed-systems problem that no current catalog federation protocol addresses.

Audit trails are an underappreciated oversight in all of this. In a single-engine environment, you can answer, "Which users accessed this sensitive table and what exactly did they read?" In a multi-engine environment with catalog federation and service accounts, the storage layer often sees only a shared IAM role that's effectively a service account. This is documented behavior: For AWS, cloud storage audit logs record the IAM principal that accessed the data, not the human user who initiated the query. There's currently no broadly adopted standard mechanism to correlate that access back to a specific user or query, which makes holistically answering a compliance audit question a build-it-yourself problem. There are ways to solve this, such as query log aggregation across platforms and/or adding trace IDs, which we will discuss in more detail in the governance interoperability deep dive post.

Snowflake's Horizon Catalog now provides audit logging for external engine access to Iceberg tables through the IRC, capturing catalog-level access events tied to authenticated identities — a meaningful step toward answering the compliance question. Correlating those catalog events back to a specific human user or query still depends on the identity model your external platforms enforce and whether they pass user-level context to the catalog. The IRC spec does not define the mechanism by which engines can send the SQL query or logic to the catalog when an engine makes a request to read a table. Therefore, if you need the SQL or the logic other engines executed for the audit log, you still need to build some log aggregation mechanism yourself. Horizon's audit logging has implemented what is possible today, and engineers working at Snowflake continue to engage with the Iceberg and Polaris communities across many features to make governance more interoperable.

Key questions to pressure-test your governance interoperability:

1. When each engine in your architecture accesses storage, does it do so exclusively through catalog-vended, short-lived scoped credentials, or do any engines require long-lived credentials that bypass the catalog entirely? And does that hold in both directions: when the catalog is the source of truth for a table and when it's the consumer for a table?
2. If a row- or column-level security policy is defined on a table in one engine's catalog, what is your mechanism for guaranteeing that another engine in your architecture enforces the same policy — and how would you detect a violation if it didn't?
3. If a compliance team asks which users accessed a sensitive table last month and what exactly they read — including access through service accounts and federated catalogs — can you answer that question with a verifiable audit trail?

There's considerably more ground to cover here than this summary allows. The deep dive goes through each FGAC approach in detail — such as scan plan API, leaf permutation materialization and trusted engine patterns. We also examine the catalog federation models and their trade-offs, approaches to cross-engine audit logging, and practical guidance for different combinations of workload attributes and compliance requirements.

Read our full deep dive on governance interoperability →

The multi-engine lakehouse is where the industry is headed, and we're committed to making it real — not just as a beneficiary of the outcome, but as an active builder of the foundation. Snowflake is a founding contributor to Apache Polaris and Apache Ossie (incubating), employing many PMC and PPMC members, respectively, to those projects. The company also employs several PMC members for Apache Iceberg. Snowflake is helping to drive specific proposals in all of these communities, aimed to close many of the gaps this series describes. We're not pointing out problems from the sidelines; we're writing the specs and shipping the implementations. That's also why we can be direct about what isn't solved yet: We're close enough to the work to know exactly where the hard edges are.

These gaps aren't permanent constraints. They're engineering problems with active proposals, real implementers and viable paths to resolution. And because the work is happening in community-governed, vendor-neutral projects, the solutions are designed to benefit the entire ecosystem — which is precisely the point.

The path from "problem recognized" to "proposed in the spec" to "implemented by all the vendors you rely on" takes time. And in the meantime, organizations wanting to go the multi-engine lakehouse route now get hurt from either the gaps themselves or having to maintain complex workarounds.

Again, it is worth reiterating that these are problems caused by simultaneous multi-engine access. Single-engine lakehouses, or even multi-engine lakehouses that don't allow simultaneous multi-engine access to a given data set, do not inherently present these issues. And simultaneous multi-engine access in your architecture isn't an all-or-nothing proposition; it can be a per-workload or per-data-set decision, based on cost-benefit analysis, which will be different for different parts of your architecture, and also change over time as these problems get solved. But this is still worth understanding, because simultaneous multi-engine access is where the ecosystem is heading. Being able to point the best engine for each job, increasingly including AI and agentic workloads, at a single copy of data without first copying it into a different system only gets more valuable over time. The point isn't to avoid it; it's to adopt it deliberately where the payoff is there today, and expand as the community solves the rest.

For large, less-sensitive analytical data sets where multiple teams legitimately need different compute tools, multi-engine access today is often worth it, and the workarounds are manageable. For curated data sets with strict compliance requirements, complex privacy policies or high-stakes business logic, the answer today is usually to store the data in Iceberg for future flexibility, while restricting all access to a single, governed engine (whether that's Snowflake or any other engine that meets your governance requirements) where your policies and logic live in one place. Open storage, curated access. Then, once the community solves the problems that apply to that data set or workload, no migration is needed. Your data is already in Iceberg format, so you can just open up additional engines to access it.

When evaluating which platform to centralize on, there's a principle that cuts through much of the noise: In an ecosystem that isn't yet fully settled, bet on platforms with a demonstrated record of genuine investment in the open standards you're building on — not just stated commitment, but verifiable participation. The community-governed nature of ASF projects like Apache Iceberg and Apache Polaris tends to signal that a platform actively co-governing the standard has its incentives structurally aligned with yours in a way that proprietary or single-vendor-driven alternatives can't replicate. This standard applies to every vendor, including Snowflake. The IRC compliance analysis is one concrete example of where stated and actual support diverge. The open source branding pattern is another; if no vendor lock-in is important to you and if a commercial offering has an open source counterpart, validate that the hosted version has meaningful parity, and apply the portability test: How easily can you move from the hosted offering to the open source one and still leverage the specific capabilities important to you? That answer tells you more than any marketing claim. Test the specific capabilities your architecture requires in your own environment, with your own data and access patterns that are representative of what you will need to do in production, before committing to any path.

The pragmatic approach: Adopt Iceberg to prevent storage lock-in, and allow simultaneous multi-engine access where the benefits outweigh the costs — but be deliberate about that call per workload or per data set. Factor in the full total cost of ownership (TCO); the people cost of implementing and maintaining workarounds is often where the real delta shows up, and it's easy to undercount in an initial architecture assessment. Reassess every six to 12 months, anchored to three things: what your specific requirements and timelines actually demand, how community progress on the open standards is tracking and what your vendors' timelines are for supporting those standards. Apache Iceberg and Apache Polaris are generally where these problems are being solved in a community-governed, vendor-agnostic way — which is exactly why they're more likely to achieve the broad adoption in engines and vendor offerings that makes the long-term bet worthwhile.

1. The interoperability challenges described in this series are inherent to the multi-engine pattern, not specific to Iceberg. Any architecture that decouples compute from storage and runs multiple SQL engines against shared data — regardless of whether that data is in Apache Iceberg, Delta Lake or Apache Hudi — face mostly the same behavioral divergences, governance gaps and business logic drift. The community has converged on Iceberg as the de facto standard table format where solutions are being built, which is why the concrete specs, proposals and workarounds in this series reference Iceberg and its ecosystem. The problems themselves are format-agnostic; most of the solutions in progress happen to live in the Iceberg community because that's where the community’s engineering investment is largely concentrated.