Operationalizing Data Interoperability in Multi-Engine Lakehouses

In discussions about multi-engine data architectures, the question that comes up most often is also the simplest: Can we have one copy of data that multiple engines efficiently and reliably read and write to without copying? For the first time in the long history of data architecture, the honest answer is "mostly yes." Apache Iceberg™ and the REST Catalog spec, along with their broad adoption, solved something that was genuinely hard five years ago. Most major engines and platforms participate as good citizens in this data architecture today.

But the more interesting question (and the one most architecture reviews skip) is: What does "mostly yes" mean, practically, when you have a specific set of engines, catalogs and workloads in production? The structural problem is solved, but some operational problems remain. Making data interoperability reliable at scale, with decentralized write pipelines and AI/ML workloads in the mix, requires (1) a concrete understanding of where the industry currently stands, (2) an assessment of trade-offs in the architectural assessment phase and (3) making decisions based on the current state and trade-offs. That's what this post covers.

If you haven't already, read the summary post, An Architect's Guide to Interoperability, which offers a brief summary of all three interoperability dimensions (data, governance and business logic interoperability) and starts to identify the gaps that exist around effective multi-engine lakehouse architectures. This post, in particular, looks comprehensively at data interoperability. It explores the concepts and guidance introduced in the summary post, plus a full analysis of workarounds and a status check on the ecosystem's progress toward that ideal vision.

Of the three interoperability dimensions this series covers, data is the furthest along — and to a significant degree. Apache Iceberg, paired with the Iceberg REST Catalog (IRC) as the access layer, has fundamentally changed how we approach data architecture. Now, you have the ability to store data once and have multiple engines read it from a shared catalog. That's meaningful and wasn't broadly true in practice five years ago.

The benefits of this community-led achievement are concrete:

• No more data duplication: No data drift, no copy pipelines, no redundant storage costs.
• Freedom to use specialized engines for different workloads: Flexibility to meet one's needs, rather than forcing everything through one tool.
• Insurance against the future: When your data is in a widely adopted open format, new engines, vendor acquisitions, pricing changes and better tools all become compute decisions, not data migrations.

Most major engines can participate today via the REST Catalog spec. That breadth of interoperability didn't exist at this scale before Iceberg.

That said, there are some important caveats worth understanding and considering. For starters, engine support for Iceberg features is uneven. Spec versions are named bundles of capabilities — a way of grouping features and managing breaking changes across the ecosystem — but support isn't always similarly bundled. Not every vendor that claims v2 support, for instance, actually supports every feature in v2 (for example, equality deletes, nan_value_counts in statistics, the data types fixed(L) and uuid), and v3 is likewise not all-or-nothing across the ecosystem. Sometimes this matters, sometimes it doesn't. Performance consistency requires active engineering discipline because Iceberg doesn't enforce file hygiene; decentralized write pipelines will produce fragmented, inconsistently configured tables that degrade quietly at scale. Disaster recovery has possible solutions today but no clean standardized path until the community ships the relative-paths proposal for v4.

There are also genuinely unsolved problems. AI/ML workload requirements (for example, wide tables, vector similarity search and point lookups) are not well-served by the current state of the Iceberg plus Parquet combination. The community knows this; proposals are in flight, but the format landscape (Lance, specialized vector databases, improvements to Parquet for wide columns) is still not settled.

For standard analytics workloads where you control the write side, multi-engine data access is achievable today with engineering discipline. For AI/ML-heavy workloads, specialized access patterns or decentralized write environments, the gaps are meaningful enough to factor into architecture decisions before you're six months in.

Let's dive into a detailed assessment and practical guide.

The REST Catalog foundation

Before the Iceberg REST Catalog spec, multi-engine data access required a separate integration for every engine-catalog pair. Every engine that wanted to read or write to Iceberg tables needed its own integration with every catalog and storage backend it might encounter (for example, Glue, Hive Metastore and Nessie), each with its own authentication model and catalog Java Archive (JAR) file. For a data platform running three engines against two storage backends, that's six integrations to build, maintain and debug independently.

The Iceberg REST Catalog (IRC) APIs collapse that integration space. Each engine integrates once with the REST Catalog API: a standardized set of endpoints and request-and-response payloads for catalog operations such as listing namespaces, loading table metadata and committing updates. Each catalog implementation exposes the same API surface. An engine that "speaks IRC" can reach any conformant catalog without modification, and any conformant catalog can serve any IRC-compliant engine. However, the Amazon S3 compatibility analogy is useful here: Storage providers frequently claim "S3-compatible" APIs without supporting every S3 endpoint, and the same dynamic applies to IRC. The spec has many endpoints, with new ones added as the standard evolves, and support for each varies across engines and catalogs. Over time, however, we believe the ecosystem should converge, and support will be more thorough, but if you expect to rely on a specific IRC capability (particularly relevant for governance APIs, which we cover here), it's important to verify support timelines across your engines and catalogs before committing to that dependency.

Apache Polaris, the open source catalog project that Snowflake co-founded, donated to the Apache Software Foundation (ASF) and is now a Top-Level Project at the ASF, implements the IRC spec, as do other major catalog implementations. Most major compute engines (for example, Apache Spark, Apache Flink, Trino, Snowflake, DuckDB and others) have IRC clients. Because Iceberg decouples data from any specific compute engine and IRC standardizes the interface between engines and catalogs, you can add or replace engines without touching the data layer or reconfiguring storage. Part of what makes this work cleanly is IRC's credential vending model: The catalog issues short-lived, scoped storage tokens to engines on demand, so engines never hold long-lived storage access independently. This foundation is what makes "structural interoperability" a fair characterization, not just an empty marketing claim. The plumbing to connect any conformant engine to any conformant catalog genuinely exists today. The caveat is that "conformant" is doing real work in that sentence. IRC compliance across the ecosystem is uneven in ways that matter, which we'll cover in a subsequent section in this deep dive.

Data type coverage

For standard analytics workloads, Iceberg's type system has been largely comprehensive since v2. The full set of primitive and nested types covers the vast majority of relational analytics use cases.

Iceberg v3 closed the most significant remaining gaps. The headline addition was a native, efficient VARIANT data type for schema-variable semi-structured data that Snowflake led and heavily contributed to. Semi-structured and nested structures can now be stored and queried natively without shoehorning them into a string column, having to define a strict schema or preshredding to a fixed schema at write time. v3 also added nanosecond-precision timestamps, geography and geometry types for spatial workloads, along with deletion vectors, a more efficient mechanism for row-level deletes. Row-lineage support rounds out the main additions, enabling each row to carry a stable identifier across updates, greatly improving CDC use cases and efficiency. For workloads beyond standard analytics (for example, AI/ML embeddings, high-dimensional search and unstructured data) the picture is different, and covered in the sections ahead.

Schema evolution and type promotion

Iceberg brings metadata-only schema evolution to the data lake: Adding, renaming, dropping or reordering columns requires no data rewrite. In a multi-engine environment, this reduces the coordination problem that would emerge when every team had to synchronize pipeline cutovers around almost every schema change. Schema changes are cheap; the risk of corrupting data during migration is eliminated; and downstream consumers aren't racing against a data rewrite in progress.

The second concrete benefit is that Iceberg brings defined, enforced type promotion rules to the data lake, something that didn't exist before. In a raw object-storage data lake, how a type change was handled depended entirely on which engine touched the data next, and different engines had different opinions. Iceberg specifies exactly which type changes are safe (for example, int to long, float to double and decimal precision widening) and enforces them at the catalog level, regardless of which engine is writing. Incompatible changes are rejected outright rather than silently producing wrong results downstream. In a multi-engine environment where multiple teams are writing to shared tables with different tools, this consistency matters.

Format and engine compatibility

The S3 compatibility dynamic from the REST Catalog section applies here too: "Supports Iceberg v2" or "supports Iceberg v3" on a release page rarely means what you'd expect. Spec version, feature completeness within that version and catalog compliance all vary independently across engines, and the gaps surface at query time rather than setup time. This piece explores v2 gaps, v3 support and how to plan for and mitigate inconsistent support across engines.

Iceberg v2 spec gaps that persist

Iceberg spec compliance is feature-by-feature, not all-or-nothing. Some v2 features still have meaningful gaps across engines today. Equality deletes are one concrete example: They're part of the v2 spec, but many engines still don't support writing them, and some don't support reading them. The reason read-side support is limited is the computational overhead. Evaluating equality delete files can require a join-like operation at scan time against every data file that is part of the table at that time, which is expensive at scale. However, in the situation of high-throughput streaming CDC writes, equality deletes are very efficient for the writer because they don't need to do any reading of any data, which is why they exist in the first place. The broader community is now moving toward deletion vectors (added in v3) and eventual index support targeted for v4 as the preferred path for streaming workload row-level deletes. However, this plan may have some problems since the indexes are currently planned to be asynchronously updated, not as part of the write commit, which means that the indexes can't be relied on to be an up-to-date picture of the table state. There have been discussions around making index updates synchronous (that is, as part of the write commit to the table, you'll also update the index structure, all as one atomic commit), but there hasn't been an appetite to go that route, at time of writing.

Pragmatic guidance

• Test equality delete support (both read and write) explicitly with your specific engine versions before relying on it in production. Also, assess whether your write engines can be configured to avoid producing equality deletes in the first place. For example, Flink currently writes equality deletes in CDC mode, but not in append mode, depending on configuration. If a write engine can't be configured away from equality deletes and your read engines don't handle them well, that's a signal to reconsider the write engine choice. If switching engines isn't an option, there are a few workaround patterns that let you keep both engines while isolating the problem:
  • Changelog + MERGE INTO: Have Flink append to a changelog table (not in CDC mode), then periodically run MERGE INTO via Snowflake or Spark to apply changes to a separate consumption table. The consumption table never has equality deletes.
  • Staging + MERGE INTO via Spark: Have Flink write in CDC mode to a staging table, then periodically run MERGE INTO via Spark to apply changes to a consumption table that your read engine points at.
  • Periodic rewrite + pause: Pause Flink writing, run rewrite_data_files in Spark with a delete file threshold of 1 to compact away equality deletes, then refresh the Iceberg table in your read engine and unpause Flink. Simple but requires a write pause window.
  • Infrequent Flink commits + timed rewrite: Configure Flink to commit infrequently (for example, every 30-60 minutes), and run rewrite_data_files in Spark with use-starting-sequence-number=false and a delete file threshold of 1 between commits, then refresh your read engine before the next Flink commit lands. This avoids a pause window but introduces unpredictable data freshness; the rewrite may not always complete in time.
  • Two-branch continuous conversion with PK index: Use Iceberg's branch support to separate write and read workloads entirely. Flink writes equality deletes to a real-time branch; a background process continuously converts them to positional deletes via PK index lookup and commits clean snapshots to a separate main branch that read-engines point at exclusively. Compactions run on the real-time branch rather than main, making them conflict-free. This is the most operationally clean approach for high-delete-rate workloads: No write pauses, no data freshness tradeoffs, and reads always see a clean snapshot. The trade-off is implementation complexity: You're building and operating the conversion process yourself.
• For tables with active row-level-delete workloads, evaluate whether deletion vectors are supported by your engines and, if so, migrate to using them over equality deletes. Snowflake supports positional deletes for v2 Iceberg tables and deletion vectors for v3 across both Snowflake-managed and externally managed tables. Watch the v4 spec progress on index support (there is an active sync on the topic in this community Google doc with the design and notes from community sync on the topic, but the Iceberg mailing list is the best place to stay up to date). Once it lands and gains engine adoption, it will further reinforce the deprecation path for equality deletes. However, it's possible that even if/when that happens, there could still be use cases that aren't solved for and need equality deletes. For example, unless the index is maintained synchronously and the index maintenance is required, the CDC writer can't rely on it for completeness to avoid writing equality deletes at scale. In this potential situation, equality deletes are still likely required for a nonnegligible amount of use cases.

v3: Patchwork support and the one-way door

For v3 specifically, the right question isn't "does this engine support v3?" It's "does this engine support the specific v3 features my workload requires?" VARIANT type support, deletion vector read and write, row lineage, nanosecond timestamps are all independently variable across engines, and Spark, Trino, Flink and others have varying levels of v3 support today. "Supports v3" in release notes often means the baseline spec is handled, not that every feature is implemented. Note that there is one strict requirement for being able to claim writing support for v3 tables, and that is writing row lineage. The Apache Iceberg spec defines the upgrade path from v2 to v3 but specifies no downgrade operation. Thus, this is the one-way door. Once a table's format-version is set to 3 in its metadata, any engine interacting with that table must handle v3 at minimum. Verify the specific features you need in each engine's documentation, not the marketing claim, because, again, there is no way to go back.

Pragmatic guidance

• Before upgrading any table to v3, map the specific features you need against the documented v3 support in every engine and catalog that touches that table, including both readers and writers.
• Upgrade tables incrementally: Start with the ones that genuinely require a v3 feature (VARIANT is a common driver), and leave remaining tables at v2 until your full engine set is ready.
• Remember: The Apache Iceberg spec defines an upgrade path to v3 but no downgrade operation. Additionally, in the Java implementation, downgrading is explicitly prohibited. This makes it critical to do testing that is representative of production workloads prior to upgrading tables to v3 in production.

Evaluating IRC implementations to avoid being locked out of the ecosystem

The concept of "bidirectional IRC compliance" is best understood from the catalog's perspective. Inbound is an engine connecting to your catalog via the IRC API to read or write your tables (that is, the catalog-as-server path). Outbound is your catalog doing federation to another catalog via IRC (that is, the catalog-as-client path, where your catalog reaches out to a remote catalog to pull in its tables). Some platforms support one direction but not the other, and the marketing claim "supports Iceberg REST Catalog APIs" washes right over that nuance.

This matters because a common multi-platform architecture involves a primary catalog that federates tables from several remote catalogs. If the remote catalog doesn't support outbound federation correctly, or the primary catalog's federation client doesn't implement the full IRC spec, you'll discover the gap at query time rather than at architecture time. As noted in Snowflake's analysis of IRC compliance across the ecosystem, stated and actual IRC support can diverge. The IRC spec does not require generation or consumption of vended credentials. Not supporting both producing and consuming vended credentials can also lead to problems with management of long-lived credentials. Claiming compliance on a feature matrix is not the same as passing the full spec against your actual workloads.

The compliance gap also has a forward-looking dimension that's easy to miss at architecture time. The IRC spec is actively evolving. The scan planning API, for instance, is a recent addition that enables server-side query planning, pushing filter and projection pushdown into the catalog rather than the engine. Future additions will go further: Read restrictions and access control enforcement at the catalog layer are on the roadmap, which would allow the catalog to govern what data an engine can see without relying on the engine to enforce it.

If your platform doesn't support full bidirectional IRC compliance today, it also can't adopt these capabilities as they land. You're locked out of the ecosystem's trajectory, not just its current state. This becomes particularly consequential in the governance space, where catalog-enforced read restrictions would eliminate a whole class of engine-trust problems.

Pragmatic guidance

• Test both inbound and outbound IRC compliance in your own environment with your own data and access patterns before committing to a catalog and engine architecture. Unfortunately, you can't rely on vendor marketing claims or documented compatibility matrices. Specifically, test with the IRC API endpoints you plan to use (including newer endpoints like the scan planning APIs and the transactions/commit endpoint) since these are the areas where support gaps are most common.
• IRC compliance is one dimension of your catalog selection decision, but weigh it against your other requirements (such as performance, governance capabilities, ecosystem support and operational overhead), based on what actually matters for your organization and workloads.

Write-side consistency

Iceberg is deliberately unopinionated about physical layout: file sizes, row group configuration, column statistics coverage, shredding strategy. That flexibility is useful, but in a multi-engine environment it means write pipelines can silently produce tables that perform well for one engine and poorly for another, with no clear messages to signal a problem. These are Parquet-layer and engine-configuration problems that apply equally to any table format built on Parquet, including Delta Lake and Hudi.

File hygiene

As introduced above, the lack of enforcement means different write pipelines operating against the same table can make different decisions about file sizes, row group sizes and statistics coverage, and what's optimal for one engine's read path can be suboptimal for another's. Without a central authority enforcing write discipline, tables can degrade gradually and silently: Query performance deteriorates, maintenance costs climb, and the degradation is hard to attribute because it doesn't emit error messages. In a centralized organization with Iceberg expertise and controlled write pipelines, this is manageable overhead. In a decentralized organization where many teams write to shared tables with different tools and varying levels of Iceberg expertise, this can quickly become a real operational liability.

For organizations where Snowflake is the primary write path, Snowflake-managed Iceberg tables offer a pragmatic approach that addresses much of this. Snowflake takes an opinionated stance on write hygiene by default: Adaptive file sizing (TARGET_FILE_SIZE = AUTO) tunes file sizes based on table characteristics rather than leaving the decision to individual pipelines; column statistics are written for all columns by default; and automatic data compaction, manifest compaction and snapshot expiry run continuously without requiring teams to schedule or manage them. The scope caveat is something to remember though: These defaults apply to Snowflake's write path. External engines writing to the same table through Horizon Catalog operate independently and don't inherit Snowflake's file-hygiene defaults — though they benefit from Snowflake's ongoing table maintenance operations on the storage side. For tables with genuinely mixed write workloads, the problem described above remains.

Pragmatic guidance

• For centralized organizations, establish explicit write pipeline standards (for example, target file size ranges, row group size and stat coverage) and enforce them in code review or pipeline templates rather than relying on individual contributors to know the defaults.
• For decentralized organizations, the more opinionated the platform, the lower the surface area for misconfiguration. Platforms that enforce sensible defaults by design significantly reduce this risk compared to giving every team direct control over write configuration.
• In both cases, run regular compaction jobs on frequently written tables to normalize file sizes, and monitor file size distributions as part of routine table health checks.

Column statistics defaults

A concrete and frequently overlooked example of write-side consistency is column statistics. Iceberg writes per-column min/max bounds, null counts and value counts into manifest metadata (that is, the statistics that query engines use to skip files that can't match a filter). The Iceberg table property write.metadata.metrics.max-inferred-column-defaults controls how many columns receive full statistics; its default is 100. For a pipeline using this default, columns beyond position 100 get no column-level statistics in Iceberg metadata at all: no min/max bounds, no null count, no value counts. The only statistics available are in the Parquet file footers, which is not as efficient as Iceberg metadata statistics for column-level predicate pushdown. A query that adds a filter on column 101, as compared to a filter on column 100, can degrade significantly without any error or warning, and the degradation only surfaces at scale when a user changes a filter predicate.

Snowflake defaults to writing full statistics for all columns because in the vast majority of cases, the query-performance benefits of having stats outweigh the costs of metadata write overhead. To be fair, there are situations where the cons outweigh the pros here, but those are generally few and far between. This is a sensible default that not all engines share.

Pragmatic guidance

• For centralized organizations, audit your table creation and write pipelines for the write.metadata.metrics.max-inferred-column-defaults setting. For tables with more than 100 columns where you filter beyond position 100, explicitly set this property to ensure full statistics are written for your actual filter columns. This is a table-level property; set it at table creation time and include it in any table provisioning standards or templates.
• For decentralized organizations, ensure that this setting is addressed by any group creating tables or writing pipelines, whether by documentation, process or technical framework.

VARIANT and shredding

VARIANT columns introduce a related but distinct problem. Iceberg v3's VARIANT type can be physically stored in Parquet with or without shredding (also known as subcolumnarization), which is the process of extracting sub-fields from the JSON structure into separate Parquet columns so that engines can benefit from columnar scan optimization on specific paths. Without shredding, querying any sub-field requires deserializing the full blob for every row. With shredding, a query on a specific path reads only that path's column. However, shredding support varies across engines, and the degree to which engines shred (how deeply they recurse into nested structures) is not standardized. Moreover, Iceberg's table metadata doesn't record the shredding strategy that was used when writing. A table written by an engine that shreds aggressively will have different physical characteristics than one written by an engine that doesn't shred at all. A user using a query engine reading both might expect them to perform equally but instead will see degraded performance with no obvious explanation.

What makes this situation particularly difficult is that the right shredding strategy is highly variable: It depends on the shape of your specific data and your query access patterns, not a static per-table rule. For simple, well-understood schemas with predictable access patterns, shredding decisions are straightforward. In production at scale (evolving schemas, mixed access patterns and/or multiple teams writing to the same tables) getting shredding right is a genuinely hard problem and ultimately presents a set of trade-offs. There are discussions ongoing in the Iceberg community around a shredding algorithm/definition.

Snowflake has spent a decade iterating on its approach to semi-structured data and shredding/subcolumnarization. That experience shows in its influence in shaping the community design for VARIANT in Iceberg v3, as well as the rubber-meets-the-road performance on VARIANT in Iceberg v3. The rest of the ecosystem is earlier on that curve.

Pragmatic guidance

• If VARIANT is a primary access pattern in your multi-engine architecture, test cross-engine query performance with representative data before production. Specifically, test queries written in each engine against data written by every other engine.
• Where possible, standardize on a single write engine for VARIANT-heavy tables to eliminate shredding variability on the write side that impacts the read side.
• For mixed-write scenarios, profile query performance per engine combination and set expectations accordingly.

Encoding and compression

Encoding and compression add a further layer of variability. Not all engines support all Parquet encoding algorithms, and the Iceberg spec doesn't mandate which encoding to use for a table. Engines tend to write what they know, which means tables in a multi-engine environment can have physically heterogeneous files: some written with one engine's preferred encoding, some with another's. This creates read-side inefficiencies when an engine encounters an encoding it handles suboptimally. One example of active standardization work: ALP (adaptive lossless floating-point) encoding for floating-point columns (contributed to the Parquet community by Snowflake engineer Prateek Gaur). Once ratified and adopted across engines, it benefits the entire ecosystem.

Pragmatic guidance

• Default to widely supported encodings for tables shared across multiple engines.
• Avoid experimental or engine-specific encodings in tables where the pros of multi-engine access outweigh the cons of degraded performance from not leveraging these advanced encodings.
• Watch the ALP vote progress for floating-point-heavy workloads.

Storage bucket management and lifecycle

Owning the storage buckets means owning everything in them, including Iceberg's own metadata. This is a materially different operational model than data warehouses, and it catches teams off-guard. With Hive, metadata lived in the catalog; the data files were blobs the catalog pointed at. With Iceberg, the metadata layer (that is, manifest files, manifest lists, table metadata JSON) lives in S3 alongside your data files. That co-location is part of what makes Iceberg fast and portable. It also means that with customer-managed storage, you're on the hook for bucket-level concerns (for example, lifecycle policies, security configuration and access controls) regardless of who manages the Iceberg metadata itself.

There's an important distinction between owning the storage and managing the Iceberg metadata operations. Even on customer-managed storage, some vendors will manage the metadata for you — running compaction, snapshot expiration, orphan cleanup and manifest rewriting as part of their service. Snowflake does this for Snowflake-managed Iceberg tables on customer-managed storage. But no vendor can override your bucket-level lifecycle policies or security settings (nor would you generally want them to). That responsibility stays with whoever owns the bucket. There is the option to go with vendor-managed storage but still store the data in Iceberg format, so other engines can access. Snowflake is one vendor that offers that in the form of Snowflake Storage for Apache Iceberg™. With that offering, you're off the ownership hook entirely: Snowflake owns the storage and manages the metadata, eliminating both layers of operational surface, while maintaining full read/write interoperability for other engines.

Without regular table maintenance (that is, orphan file cleanup, snapshot expiration, manifest rewriting and compaction) tables degrade silently, and storage costs compound. Orphan files accumulate from failed writes; old snapshots hold references to precompaction files (so you're paying double); manifest metadata bloats until query planning slows down or maintenance jobs OOM. The Slack engineering team described in its Iceberg Summit 2025 talk how a dev table receiving streaming writes every five minutes for six months, without maintenance, accumulated 12.5 TB of manifest files alone (pure metadata), and orphan file accumulation was "more prevalent than expected" across both batch and streaming workloads. If you're managing maintenance yourself (that is, not using a vendor that handles it for you), it needs to start on day one, not when performance degrades.

The most acute footgun in this space is S3 lifecycle policies applied naively. With Iceberg, metadata lives in the bucket alongside data, and lifecycle policies don't distinguish between a Parquet data file and a metadata.json. S3 lifecycle policies must be scoped to not delete any file Iceberg still references, and data deletion at the storage layer should always be driven by Iceberg's own snapshot expiration or orphan file cleanup processes, not bucket-level policies acting independently. Storage class lifecycle policies are fine to use, but only use tiers that still provide synchronous responses/access. Calculate the query performance degradations (and therefore cost) you're going to hit, and see if the pros outweigh the cons for the table and access patterns.

Pragmatic guidance

• Fix S3 lifecycle policies before anything else. Scope bucket-level deletion policies to not delete files Iceberg still references. Let Iceberg manage its own lifecycle through snapshot expiration. If you're using versioned S3 buckets, only hard-delete after Iceberg has expired the relevant snapshots.
• Start maintenance on Day 1. That is, orphan cleanup, snapshot expiration, manifest rewriting and compaction should run from the first write. Catching up on months of accumulated debt is dramatically more expensive than running it routinely. If you go with a vendor to manage this, these are usually on by default, so it's not something you have to worry about.
• Figure out your compaction and snapshot expiration strategy together. Compaction alone doesn't reclaim storage; old snapshots still reference precompaction files until expired.

Physics hasn't changed: Cross-region access and disaster recovery

Iceberg solves a lot of problems, but it doesn't repeal the laws of networking. Data stored in one region still has latency and egress costs when accessed from another. Replication still requires copying bytes across physical distance. And Iceberg's own metadata model introduces a specific structural problem that makes both cross-region access and disaster recovery harder than they were with simpler formats.

That structural problem is absolute paths. Every file reference in Iceberg metadata (that is, manifest files, manifest lists, table metadata JSON) stores the full path: s3://my-bucket/warehouse/my_table/data/part-00001.parquet, not a relative data/part-00001.parquet. If you need the same table queryable from a different location (whether for cross-region/cross-cloud environments or for business continuity and disaster recovery) you can't simply replicate the storage contents (which most cloud object storages provide out-of-the-box). The data files copy fine, but every metadata file still points to the original bucket. The replicated table is not queryable as-is without additional tooling to rewrite the paths.

According to the vote thread on the dev mailing list, the v4 spec will include a feature to support relative paths, which would eliminate this problem entirely by storing paths relative to the table's base location. It's actually a topic that's been discussed for a while in the Iceberg community, so we're glad to see it making real progress. Once v4 is ratified and adopted, you'll be able to replicate a table and simply point the catalog at the new base path. But v4 ratification, implementation across engines and broad adoption is likely at least one to three quarters out (possibly longer depending on which vendors and engines you rely on), as of this writing. It's the right long-term answer, but it isn't the answer that's available today.

The options for keeping tables queryable across regions today:

• Snowflake: Provides multiple replication options depending on how the Iceberg table is managed. For Snowflake-managed Iceberg tables, replication covers both BCDR and cross-region data sharing (via listing auto-fulfillment). Data and metadata stay consistent without custom tooling. For externally managed Iceberg tables on customer-managed storage, Snowflake supports replication for cross-region data sharing, though not BCDR in that configuration.
• AWS S3 Tables: Provide built-in replication within the AWS ecosystem, but AWS only; they don't help for cross-cloud scenarios.
• Engine-driven replication: Use a compute engine in the destination region to periodically run SQL statements (INSERT INTO ... SELECT or MERGE INTO) to materialize a local copy from the source table. This gives co-located reads in both regions at the cost of replication latency and the operational overhead of maintaining the sync pipeline.
• Build and maintain custom tooling: An initial copy with aws s3 cp (or other cloud equivalents) is straightforward; efficiently replicating ongoing changes (i.e., tracking new files, keeping metadata consistent, handling in-flight transactions) is significantly harder and becomes a piece of infrastructure to maintain indefinitely.

With that shared context, let's consider the two specific use cases more deeply.

Cross-region and cross-cloud access

The strong recommendation for any Iceberg deployment is co-location: Compute engines should run in the same cloud region as the storage they're querying. Cross-region and cross-cloud reads work. Nothing technically prevents them, but they come with real consequences:

• Egress costs on every query (outside of local engine caching queried data on disk, but that's hard to model confidently and reliably)
• Latency that degrades query performance in ways that are hard to diagnose because they look like slow engines rather than slow networks
• Increased compute costs because queries that would be fast with local storage spend time waiting on network I/O
• An expanded security surface, since data is now traversing inter-region and potentially inter-cloud network paths rather than staying within a cloud-region's trust boundary

In practice, many organizations end up in cross-region or cross-cloud situations regardless: a legacy warehouse in one region, a new Iceberg deployment in another; an acquisition that brought a different cloud footprint; an analytics team in Europe with data stored in the U.S. We see this somewhat often: A company is using two platforms in different regions without realizing it, because data is simply being copied between them. Iceberg as a project doesn't create these problems, but it doesn't solve them either. If your architecture requires cross-region or cross-cloud data access, you need to account for how you're going to solve this problem rather than assume the open format will simply absorb them.

Pragmatic guidance

• Audit your engine-to-storage topology before deploying at scale. If engines and storage are in different regions, quantify the egress cost and latency impact with representative queries on representative data volumes before committing to the architecture.
• Where possible, co-locate compute with storage. This is often the single highest-leverage configuration decision for query performance (and in the cloud, therefore cost) on large tables.
• Where co-location isn't achievable, use one of the replication approaches above to maintain a local copy, rather than accepting persistent cross-region reads at scale.

Business continuity and disaster recovery (BCDR)

BCDR requires a cross-region or cross-cloud setup almost by definition. A DR copy in the same region as your primary doesn't protect against a regional failure. The absolute paths problem described above is what makes this specifically hard for Iceberg: Standard object replication doesn't produce a queryable replica without path rewriting. Until v4 relative paths land, you need one of the solutions above to maintain a DR copy that's actually usable in a failover scenario. Also, since there are multiple levels to BCDR, if you're trying to protect against cloud failure, multi-region but single-cloud solutions don't cut it.

Pragmatic guidance

• Start by assessing which tables and workloads actually require BCDR. Not everything does, and the answer shapes the solution.
• For tables that do require BCDR, the build-versus-buy decision usually comes down to the ongoing maintenance cost. The initial copy is easy, but maintaining a consistent, queryable replica through ongoing writes, compactions and schema changes is a sustained engineering investment.
• Evaluate the options based on the most common attributes of your tables and workloads to define a default. Then, for any outlier tables or workloads, assess the options for each table/workload and see which approach is best for it.
• If you're using Snowflake Storage for Apache Iceberg and Snowflake-managed Iceberg tables, this is handled out of the box. For externally managed Iceberg tables on customer-managed storage with a near-term BCDR requirement, evaluate the engineering cost of maintaining a custom replication solution against the alternatives.

AI/ML workloads

AI/ML workloads push on the combination of Iceberg and Parquet in ways that standard analytics don't. The gaps here are real, acknowledged by the community, and actively being worked on, but not many solutions have shipped yet. This is genuinely "watch this space" territory.

The gaps fall into two kinds of issues. Some are access-pattern problems on otherwise structured data: wide feature tables and vector similarity search, where the file and index formats are the limiting factor. Others are data-type problems: unstructured and multimodal data that the table format doesn't model at all. There are a few new formats that have been created to address these problems, the main ones being Lance, Vortex and Nimble. Lance is generally more popular than the other two, so it's the one we'll discuss here.

Lance is a purpose-built lakehouse format for AI/ML, structured as a stack of interoperating specs rather than a single format: a file format (the analog to Parquet), a table format (the analog to Iceberg), plus its own index and catalog specs. It doesn't line up against just Parquet or just Iceberg. Depending on which layers you adopt, Lance can stand as an alternative to Iceberg or slot in as a complement to it. The complementary path is concrete: Iceberg's pluggable File Format API shipped in 1.11.0, decoupling table metadata from the physical file layout, and the Iceberg team explicitly names Lance (and Vortex) as formats it's meant to accommodate. A working Lance implementation on that API is still nascent. At the catalog layer you can already run both today: Apache Polaris manages Lance tables alongside Iceberg tables through its Generic Table API (not as an Iceberg file format, but as a separate table format coexisting in the same catalog). It's early, and how this shakes out isn't settled. The capabilities that make it attractive span Lance's file, index and table layers:

• Efficient vector similarity search
• Fast random access by row ID
• Native support for the column-append pattern that AI/ML feature pipelines need

Wide tables

AI/ML feature engineering routinely produces tables with hundreds or thousands of columns (for example, feature stores, embedding tables and training data sets where each feature is a column). Parquet was designed for wide reads across many rows of relatively few columns, not as much for narrow reads across a few rows of many columns. At scale, wide Iceberg tables stored in Parquet hit concrete performance problems: Parquet metadata bloats and becomes expensive to parse, row groups become small because even a modest number of rows reaches the size threshold when spread across thousands of columns, compression suffers, and adding new columns requires rewriting every data file.

The Iceberg and Parquet communities are actively exploring this problem space. A mailing list thread on "Wide tables in V4" proposed splitting wide tables across multiple physical files (column families), with early benchmarks (PR #13306) showing meaningful read and write improvements for 10,000-column tables. There's also a parallel effort in the Parquet community to replace the Thrift footer with FlatBuffers to improve parse performance for wide schemas. Whether the solution lands in Parquet, Iceberg or both is still being worked out, and either path is realistically a v4-or-later timeline.

Lance addresses this at the file and table layers with column-append that doesn't rewrite existing files (see the format note above).

Vector search and index structures

The single-copy interoperability promise doesn't extend to vector search workloads yet for broad ecosystem support. Vectors can be stored in Iceberg today, but storage isn't the problem. Efficient similarity search at scale requires dedicated index structures that Iceberg has no standard for. A community proposal was opened in 2025 but was closed as stale in January 2026 without reaching consensus. That said, there is renewed discussion in the Iceberg community to support vector indexes. The design space is genuinely hard (cross-file indexes, no agreement on format, nontrivial interaction with Iceberg's file-level abstraction), and the community hasn't converged on an approach yet.

Without native index support, vector similarity search on Iceberg tables requires a full scan or a brute-force approach that doesn't scale. The workarounds today are purpose-built vector databases for the search workload (with a pointer table in Iceberg for the structured metadata), or formats like Lance that were designed from the ground up for this access pattern.

Unstructured and multimodal data

Iceberg is a table format: It manages structured, columnar data in Parquet files (or, less frequently, in ORC and Avro files). Images, video, audio, PDFs and other unstructured data don't fit that model natively. There's no Iceberg-standard way to store a JPEG inside an Iceberg table, and forcing binary blobs into Parquet columns defeats the purpose of columnar storage (no pruning, no statistics, no compression benefit).

A usable pattern today is a pointer table. Store the raw files in object storage in their native format, and maintain a separate Iceberg table with a column pointing to each file's URI along with metadata columns (file type, creation timestamp, associated entity IDs and extracted features). This works: It's queryable, it benefits from Iceberg's schema evolution and partition pruning on the metadata, and it lets you join structured and unstructured data through the pointer. Many production systems use this pattern successfully.

The limitation is that the pointer table and the files it references are not transactionally linked:

• Deleting a row from the metadata table doesn't delete the underlying file
• Moving files in storage doesn't update the pointers
• Schema changes to the metadata don't propagate to the files

You're managing two independent systems, and keeping them consistent is operational overhead that grows with data volume.

As multimodal AI workloads grow (for example, RAG pipelines over document corpora, video analytics and image classification at scale) this gap will become an ever-widening chasm. There was an early proposal for an Iceberg unstructured data module, but it hasn't progressed beyond discussion before the issue was closed as stale in 2024. As of writing, there's no active proposal or accepted design for native unstructured data support in Iceberg, and it's unclear whether the right answer is extending Iceberg or using a purpose-built file format (Lance already supports native blob storage with lazy loading) alongside it.

Where this nets out

For workloads where vector search and point lookups are the primary access pattern, Lance has real performance advantages today that Parquet can't match without the index and wide-table work described above.

The biggest differentiating factor between Lance and Parquet is ecosystem maturity. Parquet has broad engine support, a large community and years of production-hardening across analytics workloads. Lance as a file format has a smaller ecosystem: fewer engines and less operational tooling. For mixed workloads that combine analytics and AI/ML, Parquet remains the right default. For specialized AI/ML workloads where vector search and point lookups dominate and the performance delta is significant, Lance is worth evaluating. But understand you're trading ecosystem breadth for access pattern optimization. This trade-off space is genuinely not settled, and the community's progress on wide tables and index support will narrow the gap over time.

Building on that File Format API, there's active work in the Iceberg and Lance communities to bring the Lance file format to Iceberg (for example, the foundations for future support and a POC for support). At the same time, there are discussions in the Iceberg and Parquet communities about trying to bring the benefits of Lance to Parquet. At this point, it is not yet clear how all this will shake out.

Pragmatic guidance

• Default to Iceberg and Parquet for workloads that mix analytics and AI/ML for now.
• Reach for a purpose-built format like Lance only when a specialized access pattern dominates (vector search, point lookups, column-append, or native blob/unstructured) and the performance gap justifies the ecosystem trade-off.
• For vector search specifically: If it's critical, accept the data-copy trade-off today (keep a copy in a platform with native vector search) rather than forcing Iceberg and Parquet to serve it; or, if avoiding copies is the overriding priority, store exclusively in a format that supports it natively and accept the ecosystem trade-offs.
• For unstructured and multimodal data: Use the pointer-table pattern as the default for now, and own the consistency problem explicitly (reconciliation, orphan cleanup, cascading deletes, lifecycle rules) since the table and files aren't transactionally linked.
• Where both are needed, use a unified catalog like Apache Polaris to manage Iceberg and Lance side by side rather than standardizing on one format for everything.
• Don't architect around native Iceberg support that doesn't exist yet (vector indexes, unstructured). Treat it as watch-this-space and reassess as the File Format API, wide-table and index work progresses.

If you're evaluating or already operating a multi-engine Iceberg architecture, these are the questions worth asking before you're in too deep and finding out the hard way:

1. Have you tested/validated bidirectional IRC compliance — both inbound (engines connecting to your catalog) and outbound (your catalog federating to remote catalogs) — with the specific endpoints your workloads require? Not marketing claims; actual tests with your data and access patterns.
2. Have you mapped v2 and v3 feature support per engine against your workload requirements? Specifically, do all your read engines handle the delete mechanisms your write engines produce? Do all engines that touch a v3 table support the v3 features that table uses?
3. Is all compute co-located in the same cloud region as storage? If not, have you quantified the egress cost and latency impact, and do you have an explicit strategy for keeping cross-region copies in sync?
4. Do you have a BCDR plan for the Iceberg tables that need one? Have you verified that your replication approach produces queryable replicas, given the absolute paths constraint?
5. Are your write pipelines centralized or decentralized? If decentralized, how do you enforce file size targets, column statistics coverage, VARIANT shredding consistency and encoding choices across teams and tools?
6. For tables with VARIANT columns accessed by multiple engines, have you tested cross-engine query performance with representative data? Do you know which engine wrote the data and what level of shredding it did?
7. Do any of your tables have more than 100 columns? If so, have you verified that column statistics are being written for the columns you actually filter on, or are queries silently scanning more files than necessary?
8. Do any of your write engines produce equality deletes? If so, can all your read engines handle them efficiently, or do you have a conversion or isolation strategy in place?
9. Do you have governance controls preventing storage lifecycle policies from corrupting tables? Specifically, have you verified that S3 (or equivalent) lifecycle rules can't delete any file (metadata or data) that Iceberg still references? Deleting a metadata file can make a table unreadable; deleting a data file that manifest files still point to can produce query errors on reads that hit that file, and by some definitions, corrupt the table.

Conclusion

Data interoperability is the most mature of the three pillars, but "most mature" isn't the same as "done." The gap between what works for standard analytics today and what's needed for AI/ML workloads, decentralized write environments and cross-region/cross-cloud situations is real enough to need a plan before you're six months in and paying the price for learning the hard way.

To learn more, check out An Architect's Guide to Multi-Engine Lakehouses: What's Solved and What Isn't, which is a brief summary of all three pillars, and the other deep investigations into business logic interoperability and governance interoperability.