Tools · SPBoucher
Menu

DuckDB

In-depth entry

by DuckDB Foundation / DuckDB Labs · Database · also known as Duck DB

In-process analytical SQL database that queries Parquet, CSV and Arrow data directly from Python, R, Node or the CLI

MIT licenseOpen sourceFreeLocalSelf-hostedSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

DuckDB is an open-source, in-process analytical database developed by DuckDB Labs and stewarded by the DuckDB Foundation. Like SQLite it runs inside the host process with no server to install, but it is built for analytical (OLAP) workloads: a columnar, vectorized execution engine processes large scans and aggregations quickly on a single machine.

Its most common use is as a query engine for files. DuckDB reads and writes Parquet, CSV and JSON directly, including globbed directories of partitioned files on local disk or object storage, and it exchanges data with Apache Arrow, pandas and Polars without copies. Client APIs exist for Python, R, Java, Node.js, Rust, Go and C/C++, and a standalone CLI ships for macOS, Linux and Windows. Extensions add spatial functions, full-text search, Iceberg and Delta table reading, and remote HTTP/S3 access.

In a research or finance workflow DuckDB typically sits between a Parquet data lake and a notebook or application: one SQL statement can filter a multi-year minute-bar archive, join it with a reference table and hand back an Arrow table for modelling. It is also a convenient embedded store for medium-sized applications that need SQL but not a database server.

Limits are those of a single-node embedded engine: there is no built-in multi-user server, concurrent writers to the same file are restricted, and datasets much larger than memory rely on spilling to disk, so very large or highly concurrent workloads still call for a warehouse or distributed engine. Major-version upgrades can change the on-disk storage format, so pinning versions in pipelines is advisable.

Capabilities

  • Vectorized columnar SQL execution in-process
  • Direct querying of Parquet, CSV and JSON files and directories
  • Zero-copy exchange with Apache Arrow, pandas and Polars
  • Extensions: spatial, full-text search, Iceberg/Delta, httpfs (S3)
  • Clients for Python, R, Node.js, Java, Rust, Go, C/C++ and a CLI
  • Persistent single-file databases and read-only attachments

Typical workflows

  1. Query a Parquet data lake from a notebook
  2. Replace ad-hoc pandas transformations with SQL
  3. Embed an analytical store inside a desktop or server application
  4. Convert and partition files between CSV and Parquet

Strengths

  • No server or configuration to manage
  • Excellent single-node performance on analytical scans
  • Permissive MIT license
  • First-class Arrow interoperability

Limitations

  • Single-node; no distributed execution
  • Limited concurrent write access to one database file
  • Storage format can change between major versions
  • Not designed for high-rate transactional workloads

Prerequisites

  • A host language runtime (Python, R, Node.js…) or the CLI binary

Who it is for

  • Data scientists and quantitative researchers
  • Backend developers embedding analytics
  • Data engineers building file-based pipelines

Alternatives & comparison

SQLite is the closest architectural relative (embedded, single file) but is row-oriented and tuned for transactional access; DuckDB targets analytical scans. ClickHouse offers a comparable columnar engine as a networked server with clustering. Polars covers much of the same ground as a DataFrame library rather than through SQL, and the two interoperate through Arrow.

  • Apache Arrow

    Language-independent columnar in-memory format and libraries that let DuckDB, Polars, pandas, Spark and databases exchange tables without copying

    Open sourceOpen source
  • Apache DataFusion

    Extensible Rust query engine built on Apache Arrow offering SQL and DataFrame APIs, Parquet readers and an optimizer for embedding in data systems

    Open sourceOpen source
  • Apache Iceberg

    Open table format for data lakes that adds ACID snapshots, schema and partition evolution and time travel over Parquet, ORC or Avro files

    Open sourceOpen source
  • Apache Parquet

    Open columnar file format for analytical data with compression, encodings and statistics, readable by nearly every data engine and library

    Open sourceOpen source
  • Apache Spark

    Distributed data processing engine with DataFrame and SQL APIs in Python, Scala, Java and R, plus structured streaming and MLlib

    Open sourceOpen source
  • ClickHouse

    Open-source columnar OLAP database server for fast aggregations over billions of rows, with MergeTree storage, SQL and a managed ClickHouse Cloud

    Open sourceOpen source
  • dbt

    SQL-first transformation framework that compiles templated models, tests and documentation into warehouse tables; open-source Core plus a commercial platform

    MixedOpen source
  • Delta Lake

    Open table format with a transaction log over Parquet files, created by Databricks and hosted by the Linux Foundation, with Spark, Rust and Python implementations

    Open sourceOpen source
  • GeoPandas

    Python library extending pandas with geometry columns, spatial operations, reprojection, spatial joins and file I/O for vector geodata

    Open sourceOpen source
  • H3

    Hexagonal hierarchical geospatial indexing system from Uber with a C core and bindings for Python, JavaScript, Java and SQL engines

    Open sourceOpen source
  • Kùzu

    Embedded property-graph database with Cypher, vector and full-text search; MIT-licensed but archived in October 2025 after the company was acquired

    Open sourceOpen source
  • pandas

    The standard Python DataFrame library for labeled tabular data, time series and I/O to CSV, Excel, SQL, Parquet and more

    Open sourceOpen source
  • Polars

    Rust-based DataFrame library with Python, Rust, Node and R bindings offering lazy query optimization, multithreading and Arrow-native memory

    Open sourceOpen source
  • PostGIS

    Spatial extension for PostgreSQL adding geometry, geography and raster types with indexing and hundreds of SQL functions

    Open sourceOpen source
  • PostgreSQL

    Open-source object-relational database server with a large extension ecosystem (PostGIS, pgvector, TimescaleDB) and broad SQL coverage

    Open sourceOpen source
  • SQLite

    Public-domain embedded SQL database engine that stores a whole database in one file and ships inside most operating systems and browsers

    Open sourceOpen source
  • TimescaleDB

    PostgreSQL extension for time-series data adding hypertables, compression and continuous aggregates, from the company now named Tiger Data

    MixedOpen source
  • Trino

    Distributed SQL query engine that federates queries across data lakes, warehouses and databases through connectors, formerly PrestoSQL

    Open sourceOpen source
Compare DuckDB with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Apache Arrow

Apache Software Foundation · Data catalogs & formats

In-depth entry

Language-independent columnar in-memory format and libraries that let DuckDB, Polars, pandas, Spark and databases exchange tables without copying

Apache-2.0Open sourceFreeLocalNov 2, 2026

Apache Parquet

Apache Software Foundation · Data catalogs & formats

In-depth entry

Open columnar file format for analytical data with compression, encodings and statistics, readable by nearly every data engine and library

Apache-2.0Open sourceFreeLocalNov 2, 2026

dbt

dbt Labs (Fivetran + dbt Labs) · ETL/ELT & orchestration

In-depth entry

SQL-first transformation framework that compiles templated models, tests and documentation into warehouse tables; open-source Core plus a commercial platform

Apache-2.0Open sourceFreemiumLocalSelf-hostedNov 2, 2026

Elasticsearch

Elastic N.V. · Vector databases

In-depth entry

Distributed search and analytics engine on Apache Lucene with full-text, aggregations, vector search and ES|QL; triple-licensed AGPL, SSPL and Elastic License

AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0Open sourceFreeSelf-hostedNov 2, 2026

Delta Lake

Delta Lake project (Linux Foundation; created by Databricks) · Data catalogs & formats

Open table format with a transaction log over Parquet files, created by Databricks and hosted by the Linux Foundation, with Spark, Rust and Python implementations

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

Apache Iceberg

Apache Software Foundation · Data catalogs & formats

Open table format for data lakes that adds ACID snapshots, schema and partition evolution and time travel over Parquet, ORC or Avro files

Apache-2.0Open sourceFreeSelf-hostedNov 2, 2026