Engineering Enterprise-Grade Reliability for Writes to Any Iceberg REST Catalog

Adopting an open data lakehouse with Apache Iceberg™ often means navigating a technical stack that integrates solutions from different catalog, cloud storage and database vendors. While this multi-vendor approach offers flexibility, it places a critical responsibility on the database platform. Organizations must look beyond simple query execution and choose a platform that provides enterprise-grade resilience, prevents wasteful resource usage, and seamlessly integrates security, governance, and a full suite of enterprise capabilities.

This article details the design choices for Snowflake's write path for external Iceberg tables, focusing on concurrency control, resource efficiency and commit resilience.

Key engineering challenges

Enabling reliable write operations to externally managed, Iceberg REST spec-compliant catalogs is a central part of building an enterprise-grade lakehouse. While the multi-vendor approach offers flexibility, it creates a fragmented technical stack that places a heavy responsibility on the SaaS Database system or database platform.

The fundamental difficulty stems from the fact that any SaaS database system maintains its own internal state to optimize performance, governance and security controls. In an open architecture, this internal state must be synchronized with an external catalog, which serves as the source of truth. As discussed in our previous blog regarding read operations, aligning these states is complex; however, maintaining synchronization during write operations requires solving additional concurrency and resilience problems to ensure both systems reflect the same state after a transaction.

Problem 1: Concurrency conflict and resource wastage

This challenge arises from Iceberg's reliance on optimistic concurrency control (OCC). Under this model, simultaneous commits result in a failed transaction. For enterprise workloads involving long-running operations like MERGE or UPDATE, a failure at the final commit step causes resource wastage and adds operational burden to re-execute queries. 

Problem 2: Transient network or catalog failures

Since the database does not control the external catalog, the final remote commit becomes a critical point of failure. A network issue or timeout, or a catalog service failure can interrupt the process after data files are already written, wasting the compute used to generate them. More critically, this can cause data inconsistency between the external catalog and the database platform.

Concurrency and resource efficiency

The standard Iceberg transaction protocol utilizes optimistic concurrency control OCC. Under this model, a commit conflict results in a failed transaction, requiring the query to be retried. In a multi-engine environment, however, this standard model is insufficient for high-resource workloads. For long-running mutating operations (such as MERGE or UPDATE) common in data warehousing, a query might consume substantial compute resources for an extended period, only to fail at the final commit due to a conflict. This "wasteful work" is a direct consequence of the transaction protocol, leading to unpredictable job completion times and inefficient resource expenditure.

Snowflake’s enhanced Implementation

To address this, Snowflake's implementation enhances Iceberg's optimistic transactions with a pragmatic locking mechanism, specifically for mutating DML operations (UPDATE, DELETE, MERGE). Before executing one of these queries, Snowflake acquires a lock on the table. This significantly reduces the probability of a last-minute commit conflict from other Snowflake sessions, increasing the likelihood that a resource-intensive operation will succeed on its first attempt. This lock does not apply to INSERT operations, which are append-only and can be executed in parallel without conflict.

This approach provides the transactional integrity and stability that enterprise workloads demand, directly addressing the "wasteful resource usage" challenge. While these locks govern Snowflake operations, conflicts from external engines (such as Spark) are still possible. In such cases, the engine will attempt to resolve the conflict and re-attempt the commit.

This model is complemented by a data freshness strategy that is designed for efficiency. To ensure writes are based on the most current state, Snowflake refreshes its view to the latest metadata version from the external catalog before initiating its own write operation. This ensures that sequential writes from Snowflake sessions are consistent. For standard read operations, however, Snowflake relies on the periodic AUTO_REFRESH polling mechanism. The net result is an efficient balance: Writes from other engines may have a configurable delay (up to the auto-refresh period) before becoming visible in Snowflake, which minimizes polling overhead and reduces costs associated with repeatedly checking the latest state from the catalog.

Resilience in a multi-vendor stack

The reliability of an open data lakehouse depends on the atomicity of transactions, even when interacting with external systems. In a multi-vendor stack, the database platform does not control the external catalog. This introduces a significant risk, as the final atomic "commit" call to this external service becomes a critical point of failure. A transient network issue or brief catalog unavailability events outside the database's control can interrupt the commit, causing the entire query to fail after the most resource-intensive work (writing data) is already complete.

To insulate platform users from this external fragility, an enterprise-grade failure and recovery layer is implemented in the Snowflake platform. 

First, if a transaction commit fails due to a transient external issue (e.g., a network error, catalog service failure), the system will automatically attempt to recover and reconcile the commit. This built-in resilience helps ensure data pipelines remain consistent without manual intervention. For longer-term unavailability, the system enters a protected state, temporarily blocking further writes, to prevent data inconsistency. This automated recovery is a core component of an "enterprise-grade" platform. Additionally, Snowflake runs background validation services to continuously monitor and ensure data consistency. This provides an additional layer of enterprise safety guarantees against potential inconsistencies that could arise in a multi-vendor environment with different compute engines writing to the same copy of data.

Second, the write process itself follows a clear sequence to ensure interoperability while being extensible to adding enterprise features:

• Phase 1: Data and manifest generation: The engine executes the query, writing new Parquet data files to external storage and generating the necessary Iceberg metadata (manifests and manifest lists). This phase is standard for any Iceberg-compliant engine.

• Phase 2: Atomic catalog commit (interoperable core): Once all files are written, the engine performs a single, atomic remote procedure call to the external REST catalog. This call updates the central metadata pointer to commit the new metadata.json file, finalizing the transaction. This is the standard Iceberg commit protocol, ensuring that the new state is immediately and universally visible to any other Iceberg-compliant engine, such as Spark, Trino or Flink.

• Phase 3: Snowflake metadata commit (enterprise extension): Only after the external catalog commit in Phase 2 succeeds does the engine commit any Snowflake-specific metadata (such as applied governance policies or other platform features) to its own internal state.

This layered approach is critical: It maintains 100% interoperability at its core (Phase 2) by adhering strictly to the Iceberg spec, while gracefully extending it (Phase 3) to attach Snowflake's enterprise capabilities without interfering with other engines.

Conclusion

Ultimately, enabling writes to external Iceberg tables is not just about specification compliance. It is about solving the real-world engineering challenges created by a multi-vendor data stack. The design for external writes delivers the enterprise-grade resilience and resource efficiency that organizations require from their central database platform.

This is demonstrated by:

• Concurrency control: Enhancing Iceberg's OCC with proactive locking for UPDATE, DELETE and MERGE operations to reduce wasted compute from transaction conflicts.

• Transactionality: Guaranteeing atomicity and consistency for every DML statement through a single-statement, auto-commit model.

• Layered metadata commits: Utilizing a multi-sequence commit that first adheres to the atomic Iceberg standard for universal interoperability, then commits Snowflake-specific metadata to enable enterprise features.

• Failure recovery: Implementing automated recovery mechanisms for transient external-catalog or network failures, protecting data integrity and providing true enterprise-grade resilience.