How Snowflake’s Storage Subsystem Handles Credentials Vended by Apache Iceberg Catalogs

Open data lakehouse platforms, built on formats such as Apache Iceberg, offer the flexibility to use multiple compute engines on the same data managed by an Iceberg catalog. When onboarding a new data platform like Snowflake, data architects have to make a key decision about how and where to manage the storage access for the underlying data, based on their organization’s security and governance policies. Primarily, the choice is to either manage the storage access within their database engine of their choice or centralize it in their catalog. For customers who choose to centralize governance in the catalog, our goal was to make integration seamless, reliable and secure. This required solving key engineering challenges, and this post explores how we changed Snowflake’s storage subsystem to accomplish it.

Choosing a storage credential model

When integrating an engine such as Snowflake, architects are faced with two distinct choices for managing storage access. Each choice has their valid use cases based on an organization's lakehouse setup and requirements.

Configuring credentials in the engine (via external volumes)

This model involves configuring your storage credentials directly within the database engine. A user explicitly configures and manages the credentials (like a cloud IAM role) required for Snowflake to access external cloud storage. This role is then associated with an EXTERNAL VOLUME object in Snowflake, which offers granular, engine-level control and can be ideal for organizations that prefer to manage storage access policies alongside their database configuration. However, it also means that storage security administration is split across the cloud provider's console and the database engines in the lakehouse, which can require additional scripts to keep them in sync.

The engineering challenge

Customers could have valid preferences for either model based on their lakehouse setup. The catalog-centric model offers centralized governance, but it presents a few technical hurdles: How could an engine request credentials on demand without it being slow, complex and brittle? What happens when a temporary token expires in the middle of a long-running query? What if tables within the catalog are located across many storage locations? We needed to engineer a solution that delivered the benefits of vended credentials without compromising on performance or user experience while addressing all the above challenges.

Seamless user experience with hidden volumes

The first challenge was to reduce the burden on the users to manage the creation of multiple volume objects and the volume assignment to Iceberg tables, which could become challenging as the number of tables increase. To address this issue, we changed our architecture to introduce a hidden, per-table volume. This internal volume is automatically created and managed by Snowflake for each Iceberg table and uses vended credentials. The volume’s lifecycle is tied directly to the table, which means it is created when the table is created and destroyed when it is dropped. 

Fast and resilient user queries with JIT caching

The catalog-vended credentials are scoped and temporary by nature, which ultimately leads to a critical question: What happens to a 10-hour query when its token expires after 30 minutes? To solve this problem, the hidden volume is designed for calling the external catalog's API to fetch fresh, temporary storage credentials. To enable uninterrupted long-running queries, Snowflake's caching framework monitors credential expiration and proactively calls the catalog to refresh the token before it expires. This automatic, seamless refresh mechanism manages the entire lifecycle of temporary tokens in the background, making the temporary nature of the credentials transparent to the user and the query.

Handling table creations and DMLs

When creating tables on a catalog that vends credentials, Snowflake cannot automatically determine the creation location because an external volume is not used. Consequently, the responsibility lies with either the catalog to provide a default location for table creation, or the user to explicitly specify a location or override the default using the BASE_LOCATION parameter.

For writing data files and metadata during DML and DDL operations, Snowflake relies on the storage location directly from the table's Iceberg metadata, such as the location field, as well as the table properties for correct placement of the files. This eliminates any need for manually specifying or synchronizing locations between the catalog and the database engine.

The payoff: Centralized governance and operational simplicity

Solving these core engineering challenges allows the customers who choose to centralize their storage policies in the catalog to gain operational simplicity, as well as the full platform capability of Snowflake. For data teams, this means:

• No EXTERNAL VOLUME objects to create, manage or synchronize. The setup is simplified to a single parameter ACCESS_DELEGATION_MODE in the CATALOG INTEGRATION to declare the intent to use storage credentials vended by the catalog.

• Reduced complexity in figuring out how to assign and manage lifecycle of volumes across different tables or schemas.

• Seamless storage credential scoping and refreshes for the table.

The following table highlights some of the trade-offs between choosing the database engine-managed credentials and catalog-managed credentials on the Snowflake platform: