As Snowpark adoption grew, its legacy syscall-filtering sandbox reached its design limits — some workloads needed syscalls too dangerous to allow, and maintaining allowlists wasn't scaling. We migrated to gVisor, a production-grade user-space kernel from Google. The path wasn't smooth: We hit a 500x VMA explosion and an ELF loader bug that caused segfaults in real Python packages. This post walks through what we found, how we fixed it and what the new architecture unlocked.

Limitations with legacy sandbox

Snowpark lets users write Python, Java and Scala code that runs directly inside Snowflake's virtual warehouses. To make this safe, every piece of user code runs inside a sandbox. The sandbox isolates untrusted user code from the host, prevents lateral movement and one customer's workload from affecting another's.

The original Snowpark sandbox used syscall filtering: A supervisor process maintained an allowlist of permitted syscalls and blocked everything else. For early Snowpark workloads, this was workable. But as usage expanded into diverse Data Engineering and AI/ML use cases, we're facing multiple challenges.

Popular Python packages often invoke syscalls that sit in a gray zone. Some are needed for legitimate use, while others are genuinely dangerous to expose to the host kernel due to known kernel exploits. Maintaining the allowlist became a reactive game: watch the logs, identify what failed, decide whether the syscall is safe to add, repeat. This process didn't scale.

Worse, there were cases where a workload legitimately required a syscall we couldn't safely allow at the kernel level. In those situations, the allowlist model couldn't safely accommodate the workload.

The modern architecture: gVisor + a standardized base image

To address these limitations, we chose gVisor as the foundation for the new Sandbox. gVisor is an open source, Open Container Initiative (OCI)-compatible container runtime developed by Google. Its core insight is that most kernel vulnerabilities are exploitable because user code runs in the same kernel address space as the host. gVisor flips this: Instead of running user code in the host kernel, it intercepts syscalls and reimplements most of the Linux kernel interface in a user-space process called the Sentry. The Sentry is written in Go and handles the vast majority of syscalls itself, forwarding only a small, well-controlled set to the real host kernel.

Alongside gVisor, we replaced the legacy ad hoc chroot directory with a standardized base image. At sandbox startup, the container is bootstrapped from this image, which captures the system-level libraries and binaries that the broad ecosystem of Python packages depends on. This decouples user code dependencies from whatever happens to be installed on the warehouse node, giving us a stable and portable runtime foundation.

This architecture directly addressed all three of our requirements:

• Functionality compatibility: The sandbox must support arbitrary Python packages without requiring per-workload configuration.
• Performance: Snowpark's price/performance advantage depends on low-overhead execution. The sandbox couldn't meaningfully regress.
• Maintainability: As workloads evolve, the sandbox should adapt naturally — not require manual intervention each time a new syscall pattern emerges.

The challenges: what didn't just work

Challenge 1: a 500x VMA explosion

Virtual memory areas (VMAs) are the kernel's bookkeeping structures for a process's address space. Linux caps them at 65,530 per process. Under the legacy sandbox, a typical memory-intensive Python workload created a few hundred VMAs. Under gVisor, the same workload created over 500x as many — crashing the sandbox.

The root cause was a direction mismatch in gVisor's memory allocator: It allocated virtual address space top-down but file offsets in the underlying memfd mappings bottom-up. This misalignment prevented the Linux kernel from coalescing adjacent VMAs, causing fragmentation. A secondary bug dropped the directional hint during VMA merges, making things progressively worse.

The fix aligns allocation direction and preserves the hint across merges. In internal benchmarks it reduced VMA count by 182x.

(Note: The 500x figure is the size of the original regression; the 182x is the improvement our fix delivered against the fragmented baseline. Actual VMA counts vary by workloads and this is based on an internal benchmarking of memory-intensive Python workloads, see SEE++: Evolving Snowpark Execution Environment for Modern Workloads)

Challenge 2: ELF loader incompatibility

Running workloads using the prophet package surfaced a segfault that traced back to how gVisor loaded certain ELF binaries. The ELF format distinguishes FileSiz (bytes in the file) from MemSiz (bytes in memory) — when MemSiz > FileSiz, only the delta should be zeroed. Some binaries place critical metadata like the DYNAMIC section in that padding region, expecting it to be left intact.

Meanwhile, gVisor was zeroing the full page-aligned extension unconditionally, corrupting that metadata. The fix makes gVisor's loader match Linux semantics exactly: zero only the bytes between FileSiz and MemSiz.

Challenge 3: optimizing startup latency

While migrating to gVisor was key for security and isolation, it introduced new startup performance challenges, particularly under CPU stress and high concurrency on warehouse nodes. Our profiling identified that gVisor sandbox startup, user-defined function (UDF) server startup, UDF client startup, and Python module import times were the primary bottlenecks.

To maintain Snowpark's efficiency, we implemented several key optimizations:

• gRPC server startup optimization: Starting gRPC servers for each Python worker process after forking caused significant resource contention. By moving static resource initialization earlier — using grpc_init and grpc_prefork APIs before forking — we reduced the system calls by approximately 43% for standard warehouses.
• Reducing UDF server resource consumption: We found the UDF server was consuming excess CPU through background threads for aggressive polling and stats collection. Limiting the number of available cores within the sandbox environment significantly improved overall query runtime by reducing this internal overhead.

What the modern sandbox unlocked

Beyond removing the constraints of the previous design, the architectural upgrade to gVisor directly enabled two significant Snowflake product features.

Snowpark in Serverless Tasks

Snowflake Serverless Tasks provide a fully managed, event-driven compute model. Extending this to Snowpark stored procedures and UDFs required a sandbox strong enough to safely run untrusted user code in a shared, multi-tenant environment.

The legacy syscall-filtering sandbox didn't meet that bar. gVisor's user-space syscall interception provides the stronger process isolation necessary for the multi-tenant model, making Serverless Tasks support for Snowpark Python workloads feasible.

Snowflake Artifact Repository

The Snowflake Artifact Repository lets users reference any PyPI package in Snowpark Python stored procedures and UDFs — not just a curated subset. This is a significant expansion of what Snowpark can run.

Two properties of the new architecture made this possible. First, gVisor's user-space syscall implementation means arbitrary packages can invoke the syscalls they need without us maintaining a per-package allowlist. Second, the base image provides a well-defined set of system libraries that packages can depend on, eliminating the fragile dependency on whatever happens to be present on the warehouse host.

Looking forward

The same properties that made gVisor attractive (strong isolation, full syscall coverage, a stable base image) also gave us a foundation we can build on rather than work around.

Near-term, we're continuing to expand what Snowpark can run. Further out, the same sandbox architecture that secures Python workloads today is the right foundation for the next generation of AI/ML use cases coming to Snowflake.

Disclaimer

This article contains forward-looking statements, including about our future product offerings, which are not commitments to deliver any product offerings. Actual results and offerings may differ and are subject to known and unknown risk and uncertainties.

The work described here was published as "SEE++: Evolving Snowpark Execution Environment for Modern Workloads" at IEEE BigData 2025.

Google is a trademark of Google LLC.