Tools · SPBoucher
Menu

Apache DataFusion

by Apache Software Foundation · Library · also known as DataFusion, Arrow DataFusion

Extensible Rust query engine built on Apache Arrow offering SQL and DataFrame APIs, Parquet readers and an optimizer for embedding in data systems

Apache-2.0 licenseOpen sourceFreeLocalSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

Apache DataFusion is a query engine library written in Rust and maintained as an Apache Software Foundation top-level project under Apache 2.0. It parses SQL or builds plans from a DataFrame API, optimizes them with rule-based and cost-aware passes, and executes them as multithreaded, vectorized streams of Apache Arrow record batches, reading Parquet, CSV, JSON and Avro from local disk or object stores. Every layer (parser, planner, optimizer rules, physical operators, table providers, functions) is designed to be extended or replaced, which is why DataFusion serves as the engine inside systems such as InfluxDB 3, the Comet accelerator for Spark, Ballista for distributed execution and many commercial products. Python bindings (datafusion) expose SQL and DataFrames to notebooks, and a CLI runs ad-hoc queries over files.

DataFusion is the right starting point for building a database, lake query service or domain-specific analytics tool in Rust, and a capable single-node engine for Parquet in Python. As an end-user tool it has less polish and fewer convenience features than DuckDB or Polars, and distributed execution requires additional projects.

Capabilities

  • SQL parser, planner and optimizer in Rust
  • Vectorized execution on Arrow record batches
  • Parquet, CSV, JSON and Avro readers with object-store support
  • Pluggable table providers, functions and optimizer rules
  • DataFrame API in Rust and Python
  • Command-line interface for querying files

Typical workflows

  1. Embed a SQL engine in a Rust data service
  2. Query Parquet files from Python without a database
  3. Build a custom analytics product on an extensible planner

Strengths

  • Designed for extension and embedding
  • Rust safety and performance with Arrow memory
  • Apache 2.0 under ASF governance

Limitations

  • Less turnkey than DuckDB for interactive use
  • Distributed execution needs Ballista or another layer
  • Smaller SQL function library than mature databases

Prerequisites

  • Rust toolchain or Python with the datafusion package

Who it is for

  • Developers of data systems
  • Rust engineers
  • Python users querying Parquet

Alternatives & comparison

  • Apache Arrow

    Language-independent columnar in-memory format and libraries that let DuckDB, Polars, pandas, Spark and databases exchange tables without copying

    Open sourceOpen source
  • Apache Spark

    Distributed data processing engine with DataFrame and SQL APIs in Python, Scala, Java and R, plus structured streaming and MLlib

    Open sourceOpen source
  • DuckDB

    In-process analytical SQL database that queries Parquet, CSV and Arrow data directly from Python, R, Node or the CLI

    Open sourceOpen source
  • Polars

    Rust-based DataFrame library with Python, Rust, Node and R bindings offering lazy query optimization, multithreading and Arrow-native memory

    Open sourceOpen source
  • Trino

    Distributed SQL query engine that federates queries across data lakes, warehouses and databases through connectors, formerly PrestoSQL

    Open sourceOpen source
Compare Apache DataFusion with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

  • License
    DataFusion LICENSE.txt (Apache 2.0)

    https://github.com/apache/datafusion/blob/main/LICENSE.txt

    Supports: license_spdx, source_model

    checked Nov 1, 2026
  • Official site
    Apache DataFusion

    https://datafusion.apache.org/

    Supports: name, summary, capabilities

    Site doubles as documentation portal

    checked Nov 1, 2026
  • Repository
    apache/datafusion

    https://github.com/apache/datafusion

    Supports: repo_url

    checked Nov 1, 2026
Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Apache Parquet

Apache Software Foundation · Data catalogs & formats

In-depth entry

Open columnar file format for analytical data with compression, encodings and statistics, readable by nearly every data engine and library

Apache-2.0Open sourceFreeLocalNov 2, 2026

ClickHouse

ClickHouse, Inc. · Analytical engines & OLAP

In-depth entry

Open-source columnar OLAP database server for fast aggregations over billions of rows, with MergeTree storage, SQL and a managed ClickHouse Cloud

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

dbt

dbt Labs (Fivetran + dbt Labs) · ETL/ELT & orchestration

In-depth entry

SQL-first transformation framework that compiles templated models, tests and documentation into warehouse tables; open-source Core plus a commercial platform

Apache-2.0Open sourceFreemiumLocalSelf-hostedNov 2, 2026

Elasticsearch

Elastic N.V. · Vector databases

In-depth entry

Distributed search and analytics engine on Apache Lucene with full-text, aggregations, vector search and ES|QL; triple-licensed AGPL, SSPL and Elastic License

AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0Open sourceFreeSelf-hostedNov 2, 2026

Delta Lake

Delta Lake project (Linux Foundation; created by Databricks) · Data catalogs & formats

Open table format with a transaction log over Parquet files, created by Databricks and hosted by the Linux Foundation, with Spark, Rust and Python implementations

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

Apache Iceberg

Apache Software Foundation · Data catalogs & formats

Open table format for data lakes that adds ACID snapshots, schema and partition evolution and time travel over Parquet, ORC or Avro files

Apache-2.0Open sourceFreeSelf-hostedNov 2, 2026