Tools · SPBoucher
Menu

Delta Lake

by Delta Lake project (Linux Foundation; created by Databricks) · Specification · also known as delta-rs, deltalake

Open table format with a transaction log over Parquet files, created by Databricks and hosted by the Linux Foundation, with Spark, Rust and Python implementations

Apache-2.0 licenseOpen sourceFreeSelf-hostedCloudLocalSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

Delta Lake is an open-source table format that layers an ordered transaction log (the _delta_log directory of JSON and checkpoint files) over Parquet data files to provide ACID transactions, schema enforcement and evolution, time travel and efficient upserts and deletes on object storage. It was created by Databricks, is hosted by the Linux Foundation under the Apache 2.0 license, and is the native table format of the Databricks platform, where most of its advanced features first appear. Besides the Spark connector, the delta-rs Rust library and its Python package deltalake allow reading and writing tables without a JVM, and Delta Kernel provides a connector library for other engines. Polars, DuckDB, Trino, Flink, pandas (via deltalake) and several warehouses read Delta tables; UniForm can expose a Delta table with Iceberg-compatible metadata.

Delta Lake suits teams on Databricks or Spark who need reliable incremental pipelines over a lake, and Python users who want transactional writes to Parquet from a laptop through delta-rs. Compared with Iceberg, governance of the project remains closely tied to one vendor, catalog integration is less standardized, and some features depend on the engine implementing the corresponding protocol version.

Capabilities

  • Transaction log providing ACID commits over Parquet
  • Schema enforcement and evolution
  • Time travel and table history
  • MERGE, UPDATE and DELETE operations
  • delta-rs Rust/Python implementation without a JVM
  • UniForm metadata for Iceberg readers

Typical workflows

  1. Incremental ETL with merges on Databricks or Spark
  2. Transactional Parquet writes from Python with deltalake
  3. Reading lake tables from Polars or DuckDB

Strengths

  • Simple log-based design on plain object storage
  • Mature Spark integration
  • Lightweight Rust and Python access

Limitations

  • Feature set advances first on Databricks
  • Protocol versions must be supported by each reader
  • Catalog integration less standardized than Iceberg's REST catalog

Prerequisites

  • Object or file storage plus Spark or the deltalake Python package

Who it is for

  • Spark and Databricks users
  • Python data engineers needing transactional Parquet

Alternatives & comparison

  • Apache Iceberg

    Open table format for data lakes that adds ACID snapshots, schema and partition evolution and time travel over Parquet, ORC or Avro files

    Open sourceOpen source
  • Apache Parquet

    Open columnar file format for analytical data with compression, encodings and statistics, readable by nearly every data engine and library

    Open sourceOpen source
  • Apache Spark

    Distributed data processing engine with DataFrame and SQL APIs in Python, Scala, Java and R, plus structured streaming and MLlib

    Open sourceOpen source
  • DuckDB

    In-process analytical SQL database that queries Parquet, CSV and Arrow data directly from Python, R, Node or the CLI

    Open sourceOpen source
Compare Delta Lake with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

  • License
    Delta Lake LICENSE.txt (Apache 2.0)

    https://github.com/delta-io/delta/blob/master/LICENSE.txt

    Supports: license_spdx, source_model

    checked Nov 1, 2026
  • Official documentation
    Delta Lake documentation

    https://docs.delta.io/

    Supports: capabilities, integrations

    checked Nov 1, 2026
  • Official site
    Delta Lake

    https://delta.io/

    Supports: name, summary, vendor

    checked Nov 1, 2026
  • Repository
    delta-io/delta

    https://github.com/delta-io/delta

    Supports: repo_url

    checked Nov 1, 2026
Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Apache Arrow

Apache Software Foundation · Data catalogs & formats

In-depth entry

Language-independent columnar in-memory format and libraries that let DuckDB, Polars, pandas, Spark and databases exchange tables without copying

Apache-2.0Open sourceFreeLocalNov 2, 2026

MinIO (AIStor)

MinIO, Inc. · Backup & recovery

S3-compatible object storage server; the AGPL community edition was archived in April 2026 and succeeded by MinIO's AIStor Free and AIStor Enterprise editions

MixedFreeEnterpriseSelf-hostedRenamedBroken