What Are Apache Iceberg Tables?

Iceberg is an open source table format that was originally developed by Netflix to address various challenges encountered within Apache’s Hive Hadoop project. After its initial development in 2018, Netflix donated Iceberg to the Apache Software Foundation as a completely open source, openly managed project. It remedies many of the shortcomings of its predecessor and has quickly become one of the most popular open source table formats.

The Iceberg table format offers many features to help power your data lake architecture. 

• Expressive SQL: Iceberg fully supports flexible SQL commands. This makes it possible to complete tasks such as updating existing rows, merging new data and making targeted deletes. Iceberg can be used to rewrite data files to enhance read performance and use delete deltas to quicken the pace of updates.
• Schema evolution: Iceberg supports full schema evolution. Schema updates in Iceberg tables change only the metadata, leaving the data files themselves unaffected. Schema evolution changes include adds, drops, renaming, reordering and type promotions.
• Partition evolution: Partitioning divides large tables into small ones by grouping similar rows together, speeding up read and load times for queries that only need to access a portion of the data. A partition spec can evolve without changing the earlier data written with an earlier spec. The metadata associated with each partition version is stored separately.
• Time travel and rollback: Iceberg’s time travel feature makes it possible to run reproducible queries on the same table snapshot and allows users the ability to inspect previous changes. This rollback capability allows users to easily walk back errors by resetting tables to their previous state. 
• Transactional consistency: Data stored in a data lake or data mesh architecture is available to multiple independent applications across an organization simultaneously. While this is a significant benefit, it can also come with substantial risks, especially if multiple users are writing to the same data at the same time. However, Iceberg enables ACID transactions at scale, allowing concurrent writers to work in tandem. Support for ACID helps ensure readers are not affected by partial or uncommitted changes from writers. When a writer commits a change, Iceberg creates a new, immutable version of the table’s data files and metadata.
• Faster querying: Iceberg is designed for use with huge analytical data sets. It offers multiple features designed to increase querying speed and efficiency including fast scan planning, pruning metadata files that aren’t needed, and the ability to filter out data files that don’t contain matching data.
• Vibrant community of active Apache ORCers and contributors: Iceberg is one of the Apache Software Foundation’s flagship projects. Its support for multiple processing engines and file formats including Apache Parquet, Apache Avro and Apache ORC has attracted a diverse group of talented commercial users eager to contribute to its ongoing success.