Apache Iceberg™ is the open table format for analytical workloads. Atomic writes, schema evolution, time travel and metadata-driven pruning mean an engine can skip entire files before reading a single byte. This architecture delivers fast queries on large data sets out of the box.
An Iceberg table that ingests data continuously adds files with every write. That is by design: Writes are fast and atomic because they never rewrite existing data. After a few months of streaming or micro-batch ingestion, a table can accumulate hundreds of thousands of small files, and every query starts paying a tax you didn’t sign up for.
Iceberg engines do prune files. Partition metadata and per-file column statistics allow the engine to skip files that do not match a query predicate. However, pruning has its limits. When a table is not partitioned, or when matching rows are scattered across many files rather than concentrated in a few, candidate files may not be eliminated efficiently. The engine ends up opening most of them anyway. Even when pruning works well, evaluating metadata across 1.5 million files introduces planning overhead that grows with total file count, not total data volume.
Snowflake-managed Iceberg tables address this at two complementary layers:
- Compaction consolidates small files into fewer, larger files — drastically reducing per-file and metadata overhead.
- Partitioning organizes data so the engine can eliminate entire groups of files from the scan based on partition values alone.
Neither changes the underlying data. Both fundamentally reduce how much work the engine must perform to answer the query.
The small-file problem: Why ingestion creates it and queries pay for it
When a query runs against an Iceberg table, the engine must:
- Plan: Read the metadata tree to enumerate candidate files
- Prune: Eliminate files that cannot contain matching rows using partition values, column statistics or both
- Scan: Open and read the surviving files
Each step has a per-file cost. At 500 files, you don’t notice it. At 500,000, it dominates the query, even when the answer set is tiny.
Your ingestion pipeline is creating this problem for you every day. A pipeline writing every few minutes produces thousands of files per day, each holding a narrow slice of data. After weeks or months, the metadata tree bloats, pruning loses its edge because matching rows are scattered across too many files to skip effectively and scan time is dominated by file-open overhead rather than actual data processing.
Compaction and partitioning address this at two different layers of the read path:
| Mechanism | What it reduces | How |
|---|---|---|
| Data compaction | The number of small data files and the relative file-level overhead of scanning them. | Combines small Parquet data files into fewer, larger, more efficient files while preserving the table’s data. This improves scan efficiency and query performance. |
| Partitioning | The number of data files and partitions that must be considered or scanned for a query. | Writes data into separate physical partitions based on partition transforms, such as date or region. Query predicates are used to prune irrelevant partitions before the engine performs data-file I/O. |
Compaction reduces how many files exist. Partitioning determines which ones the engine can ignore. Without partitioning, compaction still cuts file count, but the engine may still open most of them for a filtered query. Without compaction, partitioning helps with pruning, but the sheer number of files within each partition still creates overhead. You need both.
Compaction: Consolidating the file layer
Compaction rewrites many small files into fewer, larger files. The logical content of the table does not change — the same rows exist before and after — but the physical layout changes substantially.
The effect is mechanical:
- Fewer files mean less metadata to enumerate during query planning
- Larger files mean better I/O efficiency (fewer round trips, better compression ratios, more rows per read)
- Consolidated files produce tighter column statistics (min/max values), which improves predicate pushdown and file-level pruning.
Compaction runs as a background maintenance operation when Snowflake is the sole writer. For Snowflake-managed Iceberg tables, it activates automatically based on table state with no extra cost, scheduling or manual triggers required. For externally managed tables, you can enable the same optimizations, with compaction billed as a serverless compute feature.
Benchmarks are useful, but we wanted to know what compaction does for real workloads. So we looked at production telemetry for Snowflake-managed Iceberg tables and compared the same queries before and after their tables were consolidated. Same SQL, same warehouse size.
One example stood out. A customer runs the same query about 300 times a day against a frequently updated Iceberg table. After compaction the table's file count fell from roughly 104,500 files to about 490:
| Metric | Before | After |
|---|---|---|
| Files in table | ~104,500 | ~490 |
| Execution time | 3.2 sec | 1.8 sec |
| Credits per query | baseline | ~44% lower |
Compaction’s benefit scales with how fragmented your table is. Tables with frequent small writes have the most to gain. Tables already written in large batches will see smaller improvements.
Partitioning: Eliminating files before scanning
While compaction reduces how many files exist, partitioning determines how data is distributed across those files to enable multilayered pruning. At the manifest level, partition summary metadata allows the engine to skip entire manifest files containing thousands of entries without opening them. Furthermore, because data is organized by partition, min/max statistics for partition columns are far more effective for pruning, allowing the engine to eliminate irrelevant files from the scan based on metadata alone.
How partitioning works
Snowflake supports the following partition transforms:
| Transform | Effect | Example |
|---|---|---|
| Identity | Partition on the raw column value | PARTITION BY (region) |
| YEAR / MONTH / DAY / HOUR | Derive a time granularity from a timestamp column | PARTITION BY (DAY(event_time)) |
| BUCKET(N, col) | Hash-distribute into N buckets | PARTITION BY (BUCKET(16, user_id)) |
| TRUNCATE(W, col) | Truncate string or numeric to width W | PARTITION BY (TRUNCATE(3, zip_code)) |
These are hidden partitions: Users don't need to create synthetic partition columns or manage partition values manually. Iceberg derives partition values from source columns automatically, and the engine applies partition pruning transparently on queries that filter on the source column.
Partition evolution: Change your strategy without rewriting data
This is one of Iceberg’s most important capabilities and one that traditional partitioning schemes can’t match. If your query patterns change with a conventional table, you rewrite the entire thing. Iceberg eliminates that constraint.
With partition evolution, you can add, remove or change partition fields on an existing table as a metadata-only operation. Old data retains its original partition layout. New writes use the updated spec. Snowflake handles mixed partition specs transparently at query time.
-- Original table: partitioned by day only
CREATE ICEBERG TABLE events (...)
PARTITION BY (DAY(event_time));
-- Months later, query patterns now also filter heavily on region
ALTER ICEBERG TABLE events ADD PARTITION FIELD (region);Production results: 25x faster with good partitioning
Across multiple accounts at the same telecommunications company, the effect of partition pruning is visible when comparing accounts with strong vs. weak partitioning on similar Iceberg workloads:
| Account | Queries | Avg partition pruning | Avg data scanned | Avg query duration |
|---|---|---|---|---|
| Well-partitioned (Account A) | 1,406 | 75.7% files pruned | 16 GB | — |
| Well-partitioned (Account B) | 28,874 | 64.6% files pruned | 1 GB | 5.4 sec |
| Poorly partitioned (Account C) | — | 32.4% files pruned | 24 GB | 137 sec |
The poorly partitioned account scans 24x more data and runs 25x slower than the well-partitioned account — for comparable tables and query patterns. The difference is how data is physically organized relative to the predicates queries actually use.
When do these improvements help most?
These mechanisms are most relevant when a workload exhibits one or more of these characteristics:
- High file counts from streaming or micro-batch ingestion: Tables that receive frequent small writes accumulate files fastest and benefit most from compaction
- Selective queries on predictable columns: If most queries filter on the same columns, partitioning on those columns optimizes pruning
- Wide tables with narrow reads: When queries read a small subset of columns from a wide table, fewer and larger files with better compression improve column-skip efficiency
- Growing scan times on stable queries: If a query's runtime increases over weeks or months despite no query changes, file accumulation is the most likely cause
- Evolving query patterns: If the columns your queries filter on change over time, partition evolution lets you adapt without rewriting data
These mechanisms are less impactful for:
- Tables that are bulk-loaded infrequently (already have few, large files)
- Full-table scans with no predicate (nothing to prune)
Getting started
Compaction runs automatically on Snowflake-managed Iceberg tables — no configuration required. To check the state of a table:
SELECT SYSTEM$ICEBERG_TABLE_STATUS('my_db.my_schema.my_table');Partitioning is defined at table creation and can be evolved on an existing table:
-- Create a partitioned Iceberg table
CREATE ICEBERG TABLE events (
event_id STRING,
event_date DATE,
region STRING,
payload VARIANT
)
PARTITION BY (event_date, region)
CATALOG = 'SNOWFLAKE'
EXTERNAL_VOLUME = 'my_vol'
BASE_LOCATION = 'events/';
-- Evolve partitioning on an existing table
ALTER ICEBERG TABLE events ADD PARTITION FIELD (region);You can monitor compaction activity using:
SELECT *
FROM SNOWFLAKE.ACCOUNT_USAGE.ICEBERG_STORAGE_OPTIMIZATION_HISTORY
WHERE TABLE_NAME = 'EVENTS'
ORDER BY START_TIME DESC;Every Iceberg engine benefits from fewer files and better-organized partitions. The difference is who does the work. Most platforms give you a command to run, a schedule to maintain and a tuning problem that drifts as your data grows. Snowflake-managed Iceberg tables compact automatically. When your query patterns shift, partition evolution lets you adapt the physical layout without rewriting data. The result: Your Iceberg tables stay fast without you thinking about it.


