Snowpark is Snowflake's developer framework that lets engineers write data transformations, pipelines and business logic in Python, Java or Scala and run that code directly on Snowflake's virtual warehouses, without moving data out of Snowflake. Rather than pulling data into an external environment to process it, developers use a familiar DataFrame API or write user-defined functions (UDFs) that execute natively inside Snowflake's query engine, keeping data secure, governed and close to the source. As Snowpark adoption has grown and UDF workloads have become more computationally demanding, the Snowpark Execution Infrastructure team has invested heavily in runtime performance.

One of the hardest problems that surfaces at scale is data skew: the uneven distribution of work across compute nodes during query execution. Unlike traditional SQL operators, UDFs introduce highly variable per-row processing costs, which makes skew both more common and more impactful. This blog explains how we tackled that problem by building DySkew, a dynamic redistribution system that replaces our previous static, compile-time approach and has reduced P99 UDF execution times by approximately 30% in production.

Limitations with existing data skew solution

Historically, we've relied on a static round-robin redistribution strategy that we've discussed in this blog. By inserting a rebalancing link before the UDF execution operator during compile time, the system redistributes rows evenly across parallel processes. While effective for simple input skew, this approach has three fundamental limitations:

1. Compile time skew detection: We determine redistribution eligibility at compile time which means we must be conservative with the query plans we apply round-robin to as we cannot change our decision once made.
2. Computational blindness: Round-robin distribution balances row counts, not computational effort. It cannot account for variability in processing time caused by user code.
3. Poor handling of redistribution overhead: Row redistribution isn't free as we need to pack, unpack and send rows across the network. There are queries where redistribution isn't optimal, so we stored the per-row execution time of the last N invocations to determine whether redistribution was "profitable". If a function's per row execution time was below a threshold, we would assume that overhead was worse than skew and we should not apply round-robin redistribution. However, this was a binary optimization which led to performance cliffs for users when their function execution time teetered the threshold.

DySkew: Adaptive redistribution

To address these limitations, we developed DySkew, a dynamic execution strategy that moves away from static, compile-time partitioning. Instead, DySkew employs an adaptive data distribution mechanism using per-thread state machines that observe runtime conditions.

Four core principles guide the framework:

• Dynamic adaptation: Decisions are made at the link-instance level rather than globally. Each link monitors performance metrics and can transition between local processing and distributed execution in real time.
• Skew detection models: We explored several different models designed to identify skew without triggering false positives but we'll discuss two.
  • Row Percentage Model: This detects skew by monitoring whether a thread has processed significantly more rows than other threads on the same server. While this solves data skew, it doesn't solve temporal skew caused by unequal row processing times.
  • Idle Time Model: This detects skew by monitoring sibling instances. If a worker remains idle — indicating it has finished its portion of the work while others are busy — the system triggers redistribution to balance the load. This was the most successful model as it solved both data skew and temporal skew.
• N-strikes framework: To avoid overreacting to transient fluctuations, the system requires skew detection over N consecutive samples before committing to a redistribution state.
• Cost-aware redistribution: We implemented an "Eager Redistribution" strategy. For UDFs, the latency penalty for waiting to confirm skew is often higher than the cost of redistribution. By distributing early, the system prioritizes parallelism from the start of the operator's lifecycle. However, we still need to determine when redistribution can cause worse performance. Since redistribution overhead scales with row size, we introduced a Row Size Model which calculates runtime statistics, leverages the Idle Time Model for skew detection and turns off redistribution if row sizes are large while we aren't heavily skewed.

Performance and results

By moving the responsibility of skew management from compile time to execution time, we're able to make significantly better redistribution decisions which lets us apply redistribution to much more queries.

TPCx-BB

Figure 1: Performance Improvements with DySkew on TPCx-BB.

We ran the TPCx-BB benchmark on a Medium warehouse five times which confirmed significant latency reductions. There were eight queries that used UDFs and from those queries, Queries 10 and 19 showed performance improvements of 43% and 36%, respectively. At the lower end of the distribution, queries without significant skew (that is, Query 29) saw a slight regression — a deliberate tradeoff: DySkew is tuned to err toward redistributing rather than risk missing skew, since the cost of undetected skew far outweighs the overhead of occasional unnecessary redistribution.

Replayed production queries

Figure 2: Performance Improvements with DySkew on Replayed Queries.

We randomly sampled hundreds of customer production queries and replayed them with DySkew enabled vs. disabled, leading to average execution time improvement by 8% on medium and 12% on large warehouses; P90 improvement of 22% and 27%, respectively. Large warehouses saw the biggest performance gains as the likelihood and impact of skew increases with the amount of compute.

Production impact

Figure 3: Production UDxF P99 Execution Time over Time.

The above graph plots the P99 UDxF (UDF, UDAF, UDTF) execution time during rollout of DySkew. While the raw execution times are obscured, we saw an approximately 30% reduction in P99 execution times after rollout. While not pictured, we also saw CPU utilization on UDxF queries increase by approximately 17% after the rollout of DySkew.

Looking forward

DySkew demonstrates that a dynamic redistribution strategy is essential for managing the unpredictability of user-defined code in distributed systems. As we continue to refine these skew-detection models, we are exploring how to extend these adaptive capabilities to other complex query shapes, including window functions that require strict ordering. DySkew is just one example showing that Snowflake remains deeply committed to ensuring users receive the best price-performance with minimal tuning on their end.