Architecting for Governance Interoperability in Multi-Engine Lakehouses

We've had this conversation three or four times in the past month. A customer has one platform for BI and analytics, another for their ML pipelines and maybe a Spark cluster running some legacy ETL. The data team has standardized on Iceberg. They're proud of it. And then someone asks: "How do we understand data lineage across all three platforms? Or how do we enforce the same check constraints across all of them?"

In an ideal world, customers could encapsulate their governance capabilities and controls in a single catalog while mixing and matching compute engines to get the absolute best price and performance per workload. Apache Iceberg™ largely solved the storage problem and is now the de facto industry standard — any engine can read it. However, Iceberg alone doesn't know who's allowed to read it, or under what conditions, or whether it was written by an engine that respected your NOT NULL constraints.

This post is the governance deep dive from our multi-engine lakehouse series. The overview post lays out all three interoperability dimensions — data, business logic and governance. If you haven't read it, that's the right starting point. This post specifically covers governance in depth. The governance ecosystem gaps described here follow from the multi-engine access pattern itself — multiple engines reading shared data through federated catalogs — not from the choice of table format. In each section, we describe a key gap and provide a pragmatic recommendation. The guide closes with critical questions to pressure test architecture decisions.

There have been organic green shoots in this space (for example, OpenLineage, Apache Polaris™ and early Scan Plan API work). However, the tooling ecosystem around governance is still fragmented across data security, reliability and observability. Getting full coverage in a multi-engine lakehouse isn't trivial, and the good news is that it's possible. It just takes pragmatic architectural decisions and bridging the gaps between platform catalogs where no ecosystem standard yet exists (for example, classification tag propagation across catalogs and cross-platform audit log aggregation). If you're able or willing to go with a single-engine lakehouse for some workloads or tables, these gaps don't apply, and you have a simpler architecture to manage. That said, some of the solutions being advocated in the market — particularly catalog federation — don't actually solve many of these governance challenges, and in some cases, they even introduce new ones. We cover why in detail later in this post.

Data security

Object-level access: the stable ground

Table-level access control is in reasonably good shape in terms of having a practical and effective solution. The mechanism that makes this work is catalog-vended temporary credentials. Here's how they work: The catalog holds the master keys to object storage (S3, GCS, Azure Blob). When an engine requests a table, the catalog checks the user's permissions and vends a short-lived, scoped token; the engine never sees the permanent credentials and can only touch the specific files it needs for that query. This also helps reduce security risks because now there's only one long-lived credential to manage.

That said, not every Iceberg REST Catalog (IRC) and engine will support credential vending (more details here), as it was intentionally made optional as part of the Iceberg spec. For enterprise-grade deployments, however, we recommend working with vendors who have full interoperability via open security mechanisms like vended credentials.

Encryption: already pretty good, largely solved in v3

Encryption at rest with cloud-provider keys has been a nonissue for a while. Server-side encryption is a storage-layer concern: The object storage service decrypts for any authorized principal, and the engine reads plaintext bytes without ever handling a key. Because none of this lives in the table format, it works uniformly across every engine you point at the data. The limitation is what it protects against. The files are encrypted only at the storage boundary. If an engine can reach the bucket, it reads cleartext.

What v3 adds is client-side encryption, defined in the Iceberg table encryption spec. It uses envelope encryption with a three-tier key hierarchy: A table master key in your KMS wraps key-encryption-keys, which wrap a per-file data-encryption-key, all AES-GCM. Data, delete, manifest and manifest-list files are encrypted and tamper-proofed before they ever reach storage, and the wrapped keys are tracked in the table metadata's encryption-keys list. As of Iceberg 1.11, all three major CSP KMS providers are supported (that is, AWS KMS, Google Cloud KMS and Azure Key Vault). The multi-engine catch is a familiar one: Because encryption and decryption happen client-side, every engine in your stack has to implement that spec-compliant capability to read or write an encrypted table, and support is still ramping. The primitive is standardized and in the spec, but not yet implemented everywhere. For most architectures, storage-layer SSE remains the pragmatic default; reach for client-side encryption when you can't trust the storage environment and can confirm every engine that touches the table supports it.

Data observability and privacy

In a single-engine world — including single-engine lakehouses — audit logs and lineage are available out of the box in a single place. In a multi-engine lakehouse, that data is spread across multiple cloud consoles and engine logs with no built-in way to stitch it together. Catalogs can log API calls, but the engine doesn't pass the SQL or logic to the catalog to log. That fragmentation is where compliance gaps show up. Auditors ask straightforward questions (for example, "Who accessed this table last quarter and what did they do with it?"), and the answer requires pulling from three different systems that weren't designed to talk to each other. Not impossible to achieve, by any means, but not out-of-the-box.

Lineage: the OpenLineage moment

The OpenLineage framework has gained traction over the past couple of years and is being adopted by a variety of platforms (for example, Apache Airflow, Spark, dbt, Snowflake and Databricks) as the de facto way to exchange lineage information across platforms. Major platforms now natively ingest OpenLineage events, and you can see an external engine's transformation logic alongside your SQL procedures in a single end-to-end lineage view. That said, OpenLineage is a data model and API specification, not a service or router. Even with these investments, very few platforms have built the connectors to automatically exchange lineage with the others. The newly announced Horizon Context, which is part of Horizon Catalog, is something to consider because it does a lot of that collection for you. First, validate the connectors you need are available, or talk to your Snowflake team to learn more. If not going with Horizon Context, the plumbing here isn't super complicated, but it's something an organization needs to build, maintain and operate.

The identity problem in audit logs

In a single-platform architecture, this problem is largely solved. Some platforms today track the actual user, the specific tables and columns accessed, and the full SQL — regardless of whether the query came through a service account or a direct session. When everything runs through one platform, audit granularity is complete and included.

In a multi-engine setup, you're dealing with two distinct audit layers — and both have gaps.

Catalog-level audit logs record who requested access to which table and whether it was granted. That's useful, but it has limits. Most catalog audit logs capture the request (for example, "Service account X requested table Y at timestamp Z") but not the downstream detail (for example, which specific files were read, what filters were applied at the engine level or what SQL was actually executed). And critically: If your engines authenticate to the catalog via shared service accounts — which is common in federated setups — the catalog log tells you what was accessed but not who initiated the request.

Storage-level audit logs (for example, S3 CloudTrail and GCS Cloud Audit Logs) go deeper — they record every file-level access event — but at this layer, the identity problem gets worse, not better. The storage layer sees the IAM role that was used to access the files, which belongs to the catalog or the service account — not, say, Alice. Without impersonation threading the end-user identity through the full stack, you can attribute access to a service account but not a human.

This is the audit cost of a multi-compute environment. Now, there are two approaches for reducing this audit cost:

1. Trace IDs: This is the idea of attaching a unique identifier to a query that persists from the engine through the catalog down to the storage layer, letting you correlate who did what across the full stack. This is aspirational but real. Apache Polaris has merged support for including OpenTelemetry trace IDs in AWS STS session tags during credential vending, which lets you tie CloudTrail S3 access events back to a specific catalog request (detailed walkthrough here). It's an opt-in, Polaris-specific implementation today and doesn't yet extend end-to-end to the originating engine's trace context. But it's real progress toward closing this gap.

2. Centralized query log aggregation: Pull the access and query logs from each platform (query history APIs, audit logs, S3 access logs and so on) into a central table on a regular cadence. You can then run unified queries across platforms to answer, "What did Alice access last month?" without relying on any single vendor's audit tooling. Most platforms don't do this out of the box today — the plumbing to collect, normalize and correlate logs across engines is largely on customers to build and maintain themselves. A number of teams are doing it in production, and it works, but it's undifferentiated infrastructure work that the ecosystem should eventually solve natively.

Pragmatic guidance

For object-level access: Require credential vending from any catalog and engine in your architecture. It's optional in the Iceberg spec, but for enterprise deployments it should be a hard requirement. If a vendor doesn't support it, they shouldn't be in your read or write path. Snowflake's Horizon Catalog both vends short-lived scoped credentials to external engines accessing Snowflake-managed Iceberg tables, and consumes vended credentials when acting as a client to any IRC-compliant external catalog — covering both sides of the credential vending requirement in a bidirectional architecture.

For observability: Accept that you're building plumbing today. The standards exist (OpenLineage, OpenTelemetry trace IDs in Polaris), but the end-to-end integration doesn't. Two concrete actions that pay off immediately:

1. Stand up a centralized audit table: Even a simple daily ETL that pulls query logs from each platform into one place gives you a single surface for compliance questions. Don't wait for vendors to solve this for you.
2. Emit OpenLineage events from every pipeline: Even if the receiving platform doesn't auto-ingest them yet, having the events available means you're ready when it does. The cost of adding emitters now is low; the cost of retroactively reconstructing lineage is high.

For lineage: Optimally, pick one platform as your lineage aggregation point and route all OpenLineage events there. The exchange connectors often don't exist natively for all sources, but a lightweight event bus (for example, a Kafka topic or S3 bucket with event notifications) is a day of work, not a quarter. Having the events in a single place immediately gives you a usable lineage surface, regardless of which platform you're aggregating into. If Snowflake is already in your stack and/or it's your primary platform, Snowflake's Horizon Context provides end-to-end lineage aggregation — including column-level lineage across Snowflake (GA), as well as object-level lineage external databases, BI tools and OpenLineage feeds (currently in public preview) — so it's the natural aggregation point. Before committing to it, verify that the connectors you need for your specific platforms are available; coverage is broad but not yet exhaustive. If your required connectors aren't there yet, or you need a vendor-neutral approach, use the platform approach mentioned earlier in this paragraph.

Data security

Fine-grained access control: The hard part

Row- and column-level security is where things get complicated. These policies aren't standardized across platforms. If you define a row filter in your primary catalog, other platforms don't natively know how to interpret that logic.

The Apache Iceberg community has been actively working on this, with Snowflake as a key contributor. There's a formal fine-grained access control (FGAC) proposal with both a spec PR for read restrictions and a community discussion on the approach. The goal: Formalize how a catalog can evaluate policies server-side and exchange them with engines in a standard format. The work is in progress, moving through the community process though not yet merged. Until those patterns achieve broad adoption — which may be one year out or further, from the time of this writing — there are five practical approaches, each with a different cost-benefit profile.

For trusted engines: This is the long-term direction, not something that works today. The concept is that the catalog vends credentials to the full table but also sends back the mask/filter policy logic for the engine to apply locally. No materialization, no redirection. This would work for simple policies as a starting point, but it introduces real challenges. Variance in SQL dialects, function behavior and the nuances of execution nonuniformity across engines mean the same policy can produce different results depending on where it runs.

There's also a security requirement. The engine instance needs to restrict user access to the underlying nodes. Otherwise, data exfiltration via memory or spill-file access is possible. You're not trusting "Spark" or "Trino" generically — you're trusting a specific deployed instance with specific access controls in place.

Discussions are underway in the Iceberg and Apache Polaris communities to define this as a standard (the FGAC proposal and spec PR for read restrictions are being driven by contributors that include Snowflake engineers, such as Prashant Singh). There is strong consensus, and development is in progress, though broad production adoption may still be a year out; it is also dependent on vendor-implementation timelines.

Pragmatic guidance

If you're on an engine that supports the Iceberg SDK 1.11.0 or higher and your catalog supports the Scan Plan API endpoint, that's a viable path for small, low-complexity FGAC. But verify both sides of that claim before you build a dependency on it. "IRC compliant" doesn't mean every governance endpoint is implemented.

For large or complex FGAC, the simplest approach today is keeping the data in Iceberg format but restricting access for policy-protected tables to a single engine where your FGAC definitions live natively. This way, you get the storage portability of Iceberg without the policy federation headache. Open it up to multiple engines later once the community standardizes policy exchange.

On identity, it would be wise to pick a single source of truth for your data-role mappings — whether that's your identity provider (IdP), a dedicated governance tool or even a CI/CD-managed configuration repository. Most companies already have policies around keeping their IdP current, which makes it a natural choice, but the important thing is to make sure one system owns the definitions and pushes them to every catalog in your stack. If role mappings are defined independently inside each vendor's catalog, they will drift — and you won't know until an audit surfaces it.

Data classification and privacy

Data classification

Tagging a column as PII in one catalog is useless if a federated engine doesn't recognize the tag. The most reliable approach today involves generating and attaching classification tags at the point of ingest, before data enters any engine-specific catalog. That way the tags exist at the source and can be propagated downstream regardless of which engine creates derivative tables.

Pragmatic guidance

Pick a metadata source of truth: It's generally advisable to choose one platform where classification tags are mastered. It takes work to maintain parity across multiple systems manually, so before you go that route make sure you've done production-like testing and have a complete picture of the pros and cons.

Automate classification in CI/CD: If an external platform creates a derivative table, your deployment pipeline should carry PII tags to the new table in the central catalog. Manual tagging doesn't scale.

Privacy policies (aggregation, projection, clean rooms)

These are a different beast. Aggregation and projection policies — the kind you need for data clean rooms — are compute-level controls. They don't live in the table format or the storage layer. There's no standard for federating them across engines, which makes them the hardest governance primitives to share in a multi-engine setup. Today, these policies only work within the engine that defines them. Over time these types of policies need to be part of the more general policy federation across engines. However, these have a longer timeline than some of the things we discussed earlier.

Data reliability and quality

Single-engine platforms are gatekeepers. They enforce constraints at write time. In a multi-engine lakehouse platform, that gatekeeper often disappears.

Here's the failure mode: Engine A respects your NOT NULL constraint. Engine B, a legacy Spark job someone ran last Tuesday, doesn't. The data lands in storage, passes basic validation, and three weeks later, a downstream model starts returning garbage. Silent data corruption is the hardest kind of data debt to find and the most expensive to clean up. There are discussions ongoing in the Iceberg community, but this is still a work in progress.

Schema enforcement: the commit problem

Different engines have different levels of support for Iceberg table format versions and SQL dialects. A CHECK constraint defined in one platform might be completely ignored by another.

The natural solution would be catalog-level enforcement at the commit phase (that is, the catalog rejects the write if the data violates the table's schema rules, rather than just accepting whatever arrives). Today, no catalog in the open ecosystem does this comprehensively, though some platforms do have this as part of their offerings. It's a logical extension of the policy enforcement work happening in the Polaris community, but commit-time constraint validation hasn't been formally proposed as a spec item yet.

Data profiling: the stats gap

Engines don't consistently update column-level statistics — min/max values, null counts, NDVs. If Engine A writes data but skips writing stats, Engine B's query optimizer might produce a wildly suboptimal plan.

There's also no universal monitoring layer. Quality checks configured in one platform don't automatically propagate to others unless you build that integration yourself.

Pragmatic guidance

Pick a table owner: One platform is responsible for writing, monitoring, profiling and generating stats for a given data set. Don't split this responsibility across engines.

Federate constraints from a source of truth: Use a CI/CD-backed schema registry and push definitions to every catalog in your ecosystem.

The strongest link rule: If an engine in your stack can't guarantee constraint enforcement for a table, restrict it to read-only. Only engines that enforce the rules should write.

The ideal state is one universal catalog with governance and the ability to mix and match compute engines depending on the use case. This vision informs Snowflake's work with the ecosystem in both the Iceberg and Polaris communities, and how we are developing Horizon Catalog. However, the reality for most enterprises with multiple data platforms is this: Each platform needs to maintain its own source of truth.

Part of why this is hard is because historically, vendors built their compute and catalog as a tightly coupled stack. That coupling enabled real performance and governance capabilities — it wasn't accidental. But it also means decoupling is nontrivial engineering work, and progress across the ecosystem is uneven.

To their credit, vendors have been investing here. Snowflake is currently one of the few major platforms that fully support bidirectional IRC — inbound being where it acts as a server for external catalog and engines, and outbound being where it acts as a client interacting with external Iceberg REST catalogs, as well as supporting full read and write for both directions. That means external engines can read and write to Snowflake-managed Iceberg and Snowflake can read and write to external IRC compatible catalogs. Other vendors have only implemented parts of the spec — for example, Databricks Unity Catalog supports inbound IRC (that is, where it behaves like a server) read/write via Unity Catalog however doesn't support outbound IRC (that is, where it behaves like a client) and doesn't support writing to external Iceberg catalogs; AWS supports both inbound and outbound IRC for Glue but only has documented support for inbound IRC for S3 tables. Full bidirectional support with credential vending isn't universally available yet. As explained in the summary blog, the more you rely on interoperability guarantees in production, the more important IRC bidirectional compliance is.

Where full decoupling isn't yet possible, federation becomes the workaround. And federation comes with technical considerations that most teams don't discover until they're in production. For architects considering federation, it is important to understand current ecosystem gaps, the impact and potential paths to overcoming the lack of aligned standards until the community resolves these issues.

Cache and async considerations

Most current federation strategies cache metadata from a source catalog locally, close to the compute engine. While fast, it introduces two distinct problems: staleness and policy bypass.

Staleness: If a table is updated in the source, the federated engine might not see those changes for minutes or hours depending on the refresh cadence. For many analytical workloads this is acceptable. For operational workloads where freshness matters, it's not.

Policy bypass windows: If you revoke access in the source catalog, the local cache may still grant access until the next refresh cycle. In practice, the risk is bounded by the vended credential validity period. If credentials expire in 15 minutes, that's your worst-case exposure window. If the other engine doesn't support credential vending and needs long-lived credentials, this is even worse because the revokee still has access via the other engine until you also revoke the access there. This consideration informs Snowflake's strong recommendation to only platform your data based on support for full bidirectional IRC support for both access and vended credentials. For many workloads the policy bypass window within vended credentials is an acceptable risk. For regulated data with strict revocation requirements, it isn't.

The refresh model matters here. Async refresh (periodic polling) gives you stale data and the widest security bypass window. Sync or just-in-time refresh (checking the source catalog before query execution) narrows both significantly — the engine still caches locally for performance, but validates freshness against the source catalog at query time. The trade-off is latency and load on the master catalog. For data that requires low to no staleness or is security-sensitive, it's the right trade.

Snowflake's Catalog-Linked Database implementation illustrates one way to close the latency gap: Just-in-time Refresh plus Intelligent Polling (in private preview at time of writing) validates metadata against the source catalog at query time, while adaptive background polling varies the frequency, based on each table's actual change rate. When the just-in-time refresh check fires, the cache is already warm — it resolves a delta, not a cold fetch. The result is fresh reads and a narrow bypass window, without the raw latency cost that makes pure run-time catalog calls potentially problematic at scale.

Identity and policy exchange considerations

Even with just-in-time refresh solving the staleness problem, harder issues remain, and they're about identity and policies, not data freshness. The first should almost be a hard prerequisite: Get identity wrong and nothing downstream holds. The other two are federation problems where, depending on your architecture, solving just one is often sufficient:

Standardize your IdP first: This is the prerequisite that makes governance interoperability possible. If users from different platforms live in different identity systems, universal governance isn't achievable, regardless of what catalog technology you layer on top, without the complexity of building and maintaining identity mapping. You can't enforce user-level policy or produce a human-attributable audit trail through a federated path without it.

Impersonation: Most federation today authenticates via service account. The source catalog sees the service account, not the end user. Until every system in your lakehouse resolves authorization against the actual user's identity (which requires a shared IdP across all platforms), you can't enforce user-level policies through a federated path. Just-in-time refresh doesn't solve this.

Policy exchange: This is the ability for a source catalog to send not just data-access decisions, but the logic of the security policy in a format the receiving engine can enforce. This is where it gets particularly hairy. Policy logic isn't just SQL predicates. It's SQL predicates written in a specific dialect, with specific function behavior, specific null handling and specific type-coercion rules. Getting two engines to produce identical results from the same policy logic is a nontrivial problem that the ecosystem hasn't solved yet, as far as production implementation goes. This is being actively worked on in the Apache Polaris and Iceberg communities, including engineers from both Snowflake and Databricks (for example, this PR to introduce the expressions spec), but the initial scope is intentionally somewhat narrow and it is not implemented by any engine or library yet (at the time of writing). The challenge here is twofold. First, the community must agree on a mechanism for policy exchange, and then it must overcome the dialect uniformity problem.

Conclusion

Historically, compute and catalog were a package deal. You couldn't use a vendor's engine without its proprietary metadata layer. That coupling was the original walled garden — and it's what made platform migrations so expensive.

The shift, however, is happening now. Organizations can soon pick a single universal catalog as their governance source of truth, then plug in compute engines based on the best cost and performance for the workload. One catalog. Pluggable compute.

This matters for customers in three specific ways:

1. Migrations become more portable. When the catalog is independent of the compute layer, switching engines doesn't require a data migration. You simply point the new engine at the existing catalog. That said, this eliminates the storage layer of a migration, not the business logic layer. Views, UDFs, stored procedures and metric definitions still live inside specific engines and still need to be translated or rebuilt. That's its own problem — and one we cover in depth in the business logic deep dive.
2. You use the right tool for the job. Without catalog decoupling, choosing a new compute engine means creating a new data silo. With catalog decoupling, specialization is free.
3. Governance is defined once. Security policies, classification tags, lineage — defined in the catalog, enforced by every engine that reads from it.

This model is likely to be realized soon only because of the broad adoption of the Iceberg spec. By standardizing how an engine communicates with a catalog, the proprietary friction is gone. Snowflake's Horizon Catalog embeds a managed version of Apache Polaris, which means external engines can both read and write to Snowflake-managed Iceberg via standard IRC. Additionally, because we've invested in being a good citizen in the interoperable lakehouse space, Snowflake can both read and write to external IRC-compatible catalogs.

When evaluating vendors: Look for bidirectional interoperability for both read and write. Can the platform expose an IRC-compliant catalog endpoint to external engines to allow those engines to read and write, and can it consume an IRC-compliant catalog endpoint to read and write? A vendor that only exposes an IRC endpoint without supporting integration with external catalogs is still building a walled garden — they've just moved it from the storage layer to the catalog layer.

Inbound-only IRC is the faster thing to build and the more self-serving one: It lets external engines into your catalog without obligating you to interoperate with anyone else's. Full bidirectional support is more work and offers no lock-in advantage, which is exactly why it's the better signal of a vendor that intends to act as a good interoperable citizen rather than trying to leverage lock-in in order to become the new center of gravity.

If you're architecting a multi-engine lakehouse, pressure-test it against these nine questions. The ones you can't answer confidently likely expose where your actual risk sits.

The architect's governance checklist

1. The revocation gap: When you revoke a user's access to a sensitive table, how many minutes does it take for that change to propagate to every engine? Are you comfortable with that window for a compliance audit?
2. Policy drift: If a row-level security policy is defined in your primary catalog, how do you guarantee a secondary engine enforces the exact same logic?
3. Silent corruption: What do you plan to put in place to prevent an engine from writing data that violates a NOT NULL constraint defined in your source catalog?
4. Classification blind spots: When you tag a column as PII, do your other engines recognize that tag — or do they see raw data?
5. The "who" problem: Can you produce a single audit trail showing what a human user (not a service account) read last month across all tools?
6. End-to-end lineage: Does your lineage graph stop at the vendor's edge, or can you see the external engine transformation that fed the downstream table?
7. Identity standardization: Is every compute engine in your architecture connected to the same IdP?
8. The right to be forgotten: When a GDPR deletion request comes in, can you verify every derivative and cached version of that data has been purged across all catalogs?
9. Logic uniformity: If a column filter contains complex SQL logic, have you verified every engine's function behaves identically?

Most digital-native companies with large data engineering teams have built their way around these gaps — custom plumbing, dedicated headcount, internal tooling that they maintain. For the majority of organizations, though, that's not a realistic path. If you can't answer most of these questions confidently today, you're not behind — you're simply where most of the industry is.

The bottom line: Iceberg gives you everything you want at the storage layer — portability, open format, no vendor lock-in on your data. But for governance, simultaneous multi-engine access with consistent policy enforcement isn't solved yet. The community is actively working on it. It will get there. But architectures that assume it's already there will hit these gaps in production.

The data is already interoperable. The governance layer is catching up.

To learn more, check out An Architect's Guide to Multi-Engine Lakehouses: What's Solved and What Isn't, which is a brief summary of all three pillars, and the other deep investigations into data interoperability and business logic interoperability.