Databricks Unity Catalog Locks You In

Customers want choice: interoperable data governed by an interoperable catalog. Iceberg brings choice at the data layer — the promise of any engine, one copy of data, minimal switching cost. However, Unity Catalog restricts interoperability for the tables it doesn't manage, creating lock-in.

Unity Catalog claims to have an open foundation that lets any engine read and write data to tables managed by Unity Catalog, but it restricts writes to tables not managed in Unity Catalog. Databricks engines can read that data but can't write to it, forcing teams to create and maintain multiple copies of data, reconcile access across systems, and navigate complexity just to write to those tables.

The open source software (OSS) version of Unity Catalog also sits at the earliest governance tier of LF AI & Data, the Linux Foundation's umbrella for open source AI and data projects. More than two years after Databricks contributed it in June 2024, it is still at Sandbox, a stage that doesn't yet require independent governance, and Databricks remains the only established supplier of that catalog.^4,3

Yes. Unity Catalog locks in customer data with Databricks because once customers adopt Unity Catalog, switching catalogs could require reengineering their data pipelines and access controls.⁶ This happens because it restricts writes to tables customers don't manage in Unity Catalog. And once customers adopt Unity Catalog, leaving isn't easy, because customers can't easily swap in any Iceberg-compatible catalog, and even Unity Catalog OSS is not truly community-driven.³

Yes. Customers have to copy their data into Databricks to modify it because Unity Catalog can only read data managed by other catalogs, not write to it.¹ This can lead to multiple copies of data and fragmented governance.

No other vendor offers a managed version of Unity Catalog OSS.³ Customers cannot replace Unity Catalog with an independent Iceberg catalog such as Polaris or AWS Glue as the governing catalog, or use a different managed version of Unity Catalog OSS.

In license, yes; in project governance, no: Databricks owns the commercial catalog, and more than 99% of attributable commits to the OSS version come from Databricks.³

Unity Catalog OSS sits at Sandbox, the earliest of LF AI & Data's four hosting stages (Sandbox, Incubation, Graduation, Emeritus). Sandbox projects aren't yet required to have an independent Technical Steering Committee with multiple contributing organizations; that requirement only applies starting at Incubation.⁴ Databricks controls the roadmap and the contribution model, so, at this early stage, "open source" doesn't imply shared stewardship.⁷

Open format but closed catalog: To use Apache Iceberg managed or foreign tables, Databricks requires Unity Catalog, which is in turn steered by Databricks.¹³

No. Interoperability only exists when Unity Catalog is managing the tables.⁶ Two issues drive this: one-way interoperability, and security and trust risks.

No. Databricks engines can't write to foreign tables.¹ Reads go through catalog federation instead, Databricks' feature for connecting directly to an external catalog.

Catalog federation requires a long-lived storage credential set up in advance, not one issued per query, giving Databricks standing access to the customer's Iceberg storage rather than a credential scoped to a single request.⁸ This runs counter to cloud security best practice, which recommends temporary credentials over long-term ones.⁹

Turning on Unity Catalog's federation path makes foreign tables read-only. Spark itself reads from and writes to the customer's externally managed Iceberg tables via open libraries, until Unity Catalog enters the picture.¹⁰

Catalog federation gives Databricks read access to externally managed Iceberg tables, not write access. Writes through that path aren't supported.^8,11

Only for tables Unity Catalog manages, and not on Databricks workspaces using default storage.⁵ Databricks uses temporary credentials to allow other engines to read from and write to Unity Catalog-managed Iceberg tables.¹²

The reverse isn't true. To read data governed by another Iceberg catalog, customers must grant Databricks standing access to the authorized storage location (not a short-lived, request-scoped credential), which is a security and compliance risk worth evaluating because long-standing credentials may increase the risk exposure.⁸

Not a good one: Customers must copy their data into Databricks whenever they need to modify it from Databricks.¹

Catalog federation isn't a workaround either. It's Databricks' read-only path to tables governed by another catalog, so writes through that path aren't supported.^8,11

It also adds risk. Instead of getting a short-lived credential from the owning catalog for each query, catalog federation relies on a storage credential that an admin configures in advance. Databricks then reads the files directly from storage, so there's no record of the read at the owning catalog.^8,13