Managing Business Logic Interoperability in Multi-Engine Lakehouses

There is no complete, widely adopted standard for sharing business logic across engines in production. Not for lack of attempts or progress: Iceberg Views, Iceberg UDFs, Substrait and Ossie all exist in various stages of maturity. None have achieved the cross-engine adoption needed to be a reliable architectural dependency today. For anyone making production architecture decisions right now, the practical effect is the same as having no standard at all, though the seeds being planted should factor into long-term planning.

It's worth framing the scope of this problem accurately. The majority of query volume in most data platforms hits tables directly: SELECT against Iceberg tables where data interoperability (covered in the data interoperability deep dive) is sufficient. Business logic interoperability matters specifically for the workloads that route through or include views, UDFs, metric definitions and stored procedures. While that may be a smaller share of total query volume, it's where the highest-stakes business answers live. The CFO's revenue dashboard, the regulatory report, the executive KPI that drives a board decision: Those are the workloads that encode business logic, and those are the ones affected by the gaps described in this post.

The consequence is straightforward. Any metric, view, UDF or transformation logic defined in one engine must be manually redefined in every other engine that accesses the same data. There's no synchronization mechanism and no feedback loop that tells you when definitions have drifted. This isn't a hypothetical problem. Companies like Airbnb and LinkedIn have built entire internal frameworks to address it: Minerva at Airbnb for metric consistency and Coral at LinkedIn for cross-engine SQL translation. But those organizations have large dedicated engineering teams that can sustain these systems indefinitely. For most organizations, that's not a realistic path; the ongoing maintenance burden alone tends to outweigh the initial build cost.

For governance interoperability, tools like Immuta and Privacera exist to centralize policy definitions and push them to multiple engines. No equivalent exists for business logic. Semantic layer platforms like Cube and AtScale come closest: You define metrics once, and they generate engine-specific SQL at query time. But consumers must query through the tool for the consistency guarantee to hold; anything that bypasses it doesn't benefit, and the "sit-in-the-middle" architecture runs into the same scaling and single-point-of-failure problems that have historically limited data virtualization, even when they are distributed systems. Transformation tools like dbt and SQLMesh take a different approach. They generate per-engine SQL at build time from a single model definition, which is closer to "define once, deploy to many," but they only cover the transformations managed through the tool. We cover both in detail later in this post.

The problem gets worse in decentralized organizations where different business units build their own logic independently on shared data sets, each internally consistent but unaware when their definitions diverge from each other.

The pragmatic answer for today: For high-stakes business logic (for example, finance, regulatory and executive dashboards), either physicalize the logic via a pipeline that computes the result of the logic and writes the results of that computation to a table so all engines inherently get the same answer, or centralize authoritative definitions into a single, enterprise-ready engine capable of handling highly concurrent workloads. Multi-engine reads on shared Iceberg tables are reasonable for ad hoc analysis; multi-engine definitions of "recognized revenue" are not, unless the maintenance burden is worth it for a given definition. This isn't a retreat from the lakehouse vision. Your data is still in Iceberg, portable and open. You're just being deliberate about where authoritative logic lives while the ecosystem catches up.

Here's a summary table of what we'll go through in this blog:

Before evaluating the tools and specs the community has built, it's worth establishing why cross-engine business logic consistency is hard for the workloads that depend on it, which is certainly not all workloads. The behavioral differences across SQL engines are the root cause; everything that follows in this section is an attempt to work around them.

Cross-engine behavioral differences

The most fundamental challenge for business logic interoperability isn't that engines use different SQL syntax; syntax differences are well-understood and tools like sqlglot or custom tooling can translate between them. The harder problem is that engines with identical syntax produce different results because they implement SQL semantics differently. These aren't bugs; each engine is internally consistent. The problem is that "correct" is engine-specific, and two engines can both be right by their own standards while producing different answers from the same query on the same data. These divergences exist regardless of the underlying table format; they're properties of the compute engines, not the storage layer.

A few concrete examples:

• ORDER BY with null values: Spark defaults to nulls first for ascending order; DuckDB and Trino default to nulls last. A query that sorts users by last_login_date ascending to find inactive accounts will place users who have never logged in (null) at the top on Spark and at the bottom on DuckDB or Trino, changing which users appear in the first page of results.
• Type casting: Spark's behavior depends on whether ANSI mode is enabled; with ANSI mode off (the historical default; Spark 3.x defaults to ANSI mode off; Spark 4.0 changed this default, though many production deployments remain on 3.x), CAST('a' AS INT) silently returns null, while DuckDB and Trino throw an error on invalid casts by default (both provide TRY_CAST as the null-returning alternative). A pipeline that casts user input and filters on the result will silently include rows with null values on Spark (ANSI off) that would have been excluded on DuckDB or Trino.
• Timestamp handling: The timezone that an engine assumes when resolving timestamp values adds another layer. How timezone resolution interacts with session configuration and how daily or weekly aggregations behave across DST boundaries all vary.

These differences tend to surface in production on edge cases rather than in testing on clean data.

SQL is where most of the business logic portability discussion focuses, but it's not the only language in play. UDFs can be written in Python, Java or Scala with no cross-engine portability mechanism; stored procedures depend on engine-proprietary procedural languages with no equivalents elsewhere. Both are covered in dedicated sections later in this post.

The governance deep dive covers fine-grained access control (FGAC) enforcement approaches in detail. The primitive to carry forward from there is that SQL behavioral divergence can affect who sees what data, not just whether a dashboard number is right. One consequence that extends beyond analytics: FGAC policies (row filters, column masks) are typically SQL expressions. For simple predicates on standard operators (<, >=, !=), all major engines execute them the same way. But filters involving type casting, null handling or function calls enter the territory where engines can produce different results, meaning the security boundary itself varies by engine. A row filter that evaluates differently on Engine B than Engine A means users may see different data depending on which engine they query through, with no error to flag the discrepancy.

Iceberg views

With that behavioral variance as context, let's look at what the Iceberg community has built to share business logic across engines. The Iceberg view spec defines a metadata object for storing named SQL queries in the catalog alongside tables. Each view version contains one or more representations: a {type: "sql", sql: "...", dialect: "..."} tuple that records the SQL text and the dialect it was written in. A view can store multiple representations for different dialects, so in principle an engine can look for a representation matching its own dialect. The view is versioned, discoverable via the REST Catalog API and appears as a table-like object to downstream consumers.

sqlglot is an open source Python SQL parser and transpiler with support for 30+ dialects. It handles function name differences, quoting conventions, type mappings and many syntax transformations, and it emits warnings rather than silently producing wrong SQL when it can't translate cleanly. sqlglot goes beyond syntax in several important cases:

• Transpiling ORDER BY col ASC from Spark to DuckDB correctly adds an explicit NULLS FIRST clause to preserve Spark's null ordering default
• CAST('a' AS INT) from Spark (where invalid casts return null by default) becomes TRY_CAST on DuckDB (where CAST throws)
• CONCAT calls get rewritten to preserve each engine's null propagation behavior

These are genuine semantic translations, not just keyword swaps. That said, there are categories of divergence that no syntactic transpiler can address: decimal precision behavior (the same division on the same types produces different precision across engines), division-by-zero semantics (null vs. infinity vs. error) and implicit-type coercion rules that are embedded in engine runtimes rather than expressed in SQL text. For migrations, build-time SQL generation and reducing the manual work needed to stand up multi-dialect pipelines, sqlglot is a useful tool in that space. But it doesn't eliminate the need to test cross-engine equivalence for critical business logic. For high-stakes business logic, transpilation is a tool on the path to a good architecture, not a substitute for one. Authoritative definitions still need to live somewhere deliberate: a single engine or physicalized as tables.

LinkedIn's Coral takes a different approach. It is a Java-based SQL translation and rewrite engine that uses an intermediate representation (based on Apache Calcite's relational algebra) to translate between HiveQL, Spark SQL and Trino dialects. LinkedIn operates it in production across its data platform, demonstrating that IR-based translation works at large data volumes for a specific set of dialects, particularly when maintained by a large engineering organization. The limitation, however, is that the supported dialect set is narrow, and extending it requires deep familiarity with Calcite's framework. While Coral is open source, it's not a turnkey solution for most teams.

The Substrait project is working toward a broader version of this concept: a cross-language specification for serializing relational algebra that any engine could produce or consume. If business logic were stored as Substrait plans rather than SQL text, dialect differences would be handled by engine-specific adapters rather than text translation. In practice, adoption has been uneven. Apache Arrow's Acero and DataFusion (as a producer and consumer), and Ibis (as a producer) have native Substrait support, but the major query engines that dominate Iceberg lakehouse deployments don't have native Substrait integration. Until the engines you actually use support it, Substrait is a promising architecture rather than a production dependency.

Within Iceberg itself, the recently merged expressions spec (Iceberg PR #16652) is a narrower take on the same idea. Instead of raw SQL text, it represents expressions as standardized expression signatures. The structure is engine-agnostic, but the spec deliberately stops there. It doesn't say how the defined expressions actually behave, leaving that to each engine. It standardizes the shape of an expression, not its implementation behavior. That makes it foundational plumbing (scalar expressions, not full queries) that view and UDF definitions could eventually build on, but not a fix for the behavioral divergence described above.

A fourth, semantic-layer-level variant is on Ossie's roadmap: a reference engine plus an Ossie-to-SQL compiler and a cross-implementation conformance suite, so a single semantic model definition could be compiled to each engine's dialect with consistent interpretation. It is scoped to Ossie-modeled semantic queries (not arbitrary SQL, views or UDFs) and is on the roadmap rather than shipped, but it applies the same canonical-representation idea at the semantic layer.

There's also an accountability problem that applies to all transpilation and IR approaches. In a single-engine world, when a query produces the wrong answer, there's one vendor to call. In a multi-engine world with a translation layer in between, a wrong result could be the fault of the source engine, the target engine or the translation layer. Debugging cross-engine result differences becomes a cross-vendor support problem: slow to resolve, ambiguous in ownership and costly when it surfaces in a downstream business decision. The governance deep dive covers why this accountability gap is particularly consequential for security policies.

Pragmatic guidance

sqlglot covers more ground than most teams expect: Use it for migrations, build-time SQL generation and as a first pass when standing up multi-dialect pipelines. For the divergences it handles (null ordering, type casting, null propagation in functions), it genuinely reduces the manual work. For the divergences it can't (precision, division-by-zero, implicit coercion), cross-engine regression tests on representative data are the backstop. Watch Substrait adoption across the specific engines in your architecture; it's a sound long-term concept but not yet a dependency you can rely on. For critical business logic, the architectural guidance from the current state assessment still applies: Centralize authoritative definitions in a single engine or physicalize the results rather than depending entirely on a translation layer.

Stored procedures and procedural logic

Unlike views, UDFs and metrics, stored procedures don't have an Iceberg spec equivalent, a community proposal or even a widely discussed path forward. This isn't an oversight; it's a reflection of how deeply engine-coupled stored procedures are. Views and UDFs are at least SQL expressions that can be stored and potentially translated. Stored procedures are full programs: loops, conditionals, exception handling, transaction control, cursor management, dynamic SQL and calls to engine-specific system functions, all written in engine-proprietary procedural languages (PL/pgSQL, Snowflake Scripting, Spark's programmatic APIs). There's no shared abstraction because there's no shared surface area to abstract over.

In a single-engine world, stored procedures are where critical multi-step business workflows live: end-of-month financial close processes, data quality enforcement routines, complex ETL orchestration, conditional transformation logic that can't be expressed as a single SQL statement. In a multi-engine world, these are entirely engine-locked with no portability path. Transpilation tools like sqlglot operate on SQL expressions, not procedural control flow. Intermediate representations (IRs), like Substrait, model relational algebra, not imperative logic. Even if every other category of business logic described in this post achieved full cross-engine portability tomorrow, stored procedures would remain tied to the engine they were written for.

Stored procedures are also where complex governance logic sometimes lives: multi-step anonymization workflows, conditional access decisions that go beyond what a single-row filter expression can handle, audit logging routines, to name a few. When that logic is engine-locked, the compliance implications extend beyond analytics consistency into security posture; the governance deep dive covers this in detail.

Pragmatic guidance

Accept that stored procedures are engine-locked and plan accordingly. If a critical business workflow lives in a stored procedure, that's one of the strongest arguments for routing that workload through a single engine. Where possible, refactor procedural logic into declarative SQL (which has a portability path, even if an imperfect one) or externalize the orchestration layer to tools like Airflow, Dagster or Prefect, which coordinate workflow steps without depending on engine-specific procedural languages.

The remaining long tail

Even if views, UDFs, metrics and stored procedures all had cross-engine solutions tomorrow, a set of lower-priority business logic categories would remain engine-specific. Triggers and event-driven logic have no portability mechanism and no community proposal addressing them. Scheduled tasks and job orchestration defined inside an engine (as opposed to external orchestrators like Airflow) are tied to that engine's scheduler semantics. Materialized view refresh logic and maintenance schedules are engine-specific by definition. Session state and context functions used in business logic (CURRENT_USER(), CURRENT_ROLE(), session variables) return engine-specific values with no cross-engine equivalence guarantee, which matters when those functions appear in access control policies or audit logic. None of these are typically the source of the "two dashboards, different numbers" problem, which is why they're lower priority than the categories covered earlier. But they are worth acknowledging: Any architecture assessment that only accounts for views, UDFs and metrics will undercount the total surface area of engine-coupled business logic.