Tools · SPBoucher
Menu

Apache Iceberg

by Apache Software Foundation · Specification · also known as Iceberg, PyIceberg

Open table format for data lakes that adds ACID snapshots, schema and partition evolution and time travel over Parquet, ORC or Avro files

Apache-2.0 licenseOpen sourceFreeSelf-hostedCloudSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

Apache Iceberg is an open table format maintained by the Apache Software Foundation under Apache 2.0. It defines metadata (table metadata files, manifest lists and manifests) that tracks which Parquet, ORC or Avro data files make up each snapshot of a table, so that engines can commit atomic changes, read a consistent snapshot, travel back in time, evolve schemas and change partitioning without rewriting data. Hidden partitioning lets queries filter on raw columns while the layout stays an implementation detail. The REST catalog specification standardizes how engines discover and update tables, and catalogs such as Apache Polaris, Nessie, AWS Glue and vendor offerings implement it.

Iceberg is supported by Spark, Trino, Flink, DuckDB, ClickHouse, Snowflake, BigQuery, Athena, Dremio and PyIceberg, the Python client used to read tables into Arrow or write small batches. It is the usual answer when several engines must share one governed copy of lake data. The cost is metadata and catalog operations to run and maintain, compaction and snapshot expiration jobs to schedule, and the fact that single-user analytics on local Parquet rarely needs it.

Capabilities

  • Atomic snapshots and serializable commits over object storage
  • Schema evolution and partition evolution without rewrites
  • Hidden partitioning and time travel
  • Manifest-based file pruning with column statistics
  • REST catalog specification
  • Support in Spark, Trino, Flink, DuckDB, ClickHouse and PyIceberg

Typical workflows

  1. Shared lakehouse tables read by Trino and written by Spark
  2. Incremental ingestion with snapshot-based change reads
  3. Reading a governed table from Python via PyIceberg and Arrow

Strengths

  • Vendor-neutral format with broad engine support
  • Transactions and time travel on plain object storage
  • Partition changes without data migration

Limitations

  • Requires a catalog service and maintenance jobs
  • Overkill for single-user local analytics
  • Write support in lightweight engines is still partial

Prerequisites

  • Object or file storage plus a catalog; an engine or PyIceberg client

Who it is for

  • Data platform engineers
  • Analytics teams sharing lake data across engines

Alternatives & comparison

  • Apache Parquet

    Open columnar file format for analytical data with compression, encodings and statistics, readable by nearly every data engine and library

    Open sourceOpen source
  • Apache Spark

    Distributed data processing engine with DataFrame and SQL APIs in Python, Scala, Java and R, plus structured streaming and MLlib

    Open sourceOpen source
  • Delta Lake

    Open table format with a transaction log over Parquet files, created by Databricks and hosted by the Linux Foundation, with Spark, Rust and Python implementations

    Open sourceOpen source
  • DuckDB

    In-process analytical SQL database that queries Parquet, CSV and Arrow data directly from Python, R, Node or the CLI

    Open sourceOpen source
Compare Apache Iceberg with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

  • License
    Iceberg LICENSE (Apache 2.0)

    https://github.com/apache/iceberg/blob/main/LICENSE

    Supports: license_spdx, source_model

    checked Nov 1, 2026
  • Official documentation
    Iceberg documentation

    https://iceberg.apache.org/docs/latest

    Supports: capabilities, integrations

    checked Nov 1, 2026
  • Official site
    Apache Iceberg

    https://iceberg.apache.org/

    Supports: name, summary

    checked Nov 1, 2026
  • Repository
    apache/iceberg

    https://github.com/apache/iceberg

    Supports: repo_url

    checked Nov 1, 2026
Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Apache Arrow

Apache Software Foundation · Data catalogs & formats

In-depth entry

Language-independent columnar in-memory format and libraries that let DuckDB, Polars, pandas, Spark and databases exchange tables without copying

Apache-2.0Open sourceFreeLocalNov 2, 2026

MinIO (AIStor)

MinIO, Inc. · Backup & recovery

S3-compatible object storage server; the AGPL community edition was archived in April 2026 and succeeded by MinIO's AIStor Free and AIStor Enterprise editions

MixedFreeEnterpriseSelf-hostedRenamedBroken