Tools · SPBoucher
Menu

Apache Arrow

In-depth entry

by Apache Software Foundation · Specification · also known as Arrow, PyArrow, Arrow Flight, ADBC

Language-independent columnar in-memory format and libraries that let DuckDB, Polars, pandas, Spark and databases exchange tables without copying

Apache-2.0 licenseOpen sourceFreeLocalSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

Apache Arrow is an Apache Software Foundation project that defines a language-independent columnar memory format for tabular data together with libraries that implement it in C++, Python (PyArrow), Rust, Java, Go, C#, JavaScript, R, Ruby, Swift and others. Because every implementation lays out arrays identically in memory, systems can hand each other tables by pointer rather than by serializing and parsing, which is why Arrow has become the interchange layer between DuckDB, Polars, pandas, Spark, DataFusion, Snowflake and many databases and file readers.

Beyond the format, the project specifies the Arrow IPC stream and file formats for moving record batches over sockets or disk, the Arrow Flight RPC protocol and Flight SQL for high-throughput data services, ADBC as a columnar replacement for ODBC and JDBC drivers, and the C Data Interface for zero-copy sharing between libraries in the same process. The C++ library includes Parquet, CSV, JSON and ORC readers and writers, dataset scanning over partitioned directories and object stores, and the Acero compute engine; PyArrow exposes these to Python and backs the Arrow-native dtypes in pandas.

For a researcher the practical effect is that a Parquet archive read with PyArrow can be queried by DuckDB, transformed by Polars and plotted from pandas with few or no conversions. Arrow is released under Apache 2.0. Its limits are those of a format and toolkit rather than a product: there is no server or storage to deploy, in-memory layouts favor analytical scans over row-at-a-time updates, and library versions across languages must be kept compatible when sharing data through the C interface.

Capabilities

  • Standardized columnar in-memory layout for tables and nested types
  • IPC stream and file formats for record batches
  • Arrow Flight and Flight SQL RPC protocols
  • ADBC database connectivity API
  • C Data Interface for zero-copy in-process sharing
  • Parquet, CSV, JSON and ORC readers and writers (C++/PyArrow)
  • Dataset API for partitioned files on local disk and object storage
  • Implementations in more than ten languages

Typical workflows

  1. Read a Parquet lake with PyArrow and hand tables to DuckDB or Polars
  2. Serve query results over Arrow Flight to analysts
  3. Replace ODBC/JDBC with ADBC drivers for columnar fetches
  4. Exchange DataFrames between Python, R and Rust processes

Strengths

  • Eliminates serialization between analytical tools
  • Apache 2.0 and foundation governance
  • Reference implementations in many languages
  • Widely adopted as the de facto interchange standard

Limitations

  • A format and library set, not a database or server
  • Columnar layout is poor for row-level updates
  • Cross-language version compatibility must be managed
  • PyArrow wheels are large and platform-specific

Prerequisites

  • A language runtime with an Arrow implementation (e.g. pip install pyarrow)

Who it is for

  • Data engineers
  • Quantitative researchers
  • Developers of databases and data tools

Alternatives & comparison

Apache Parquet is the on-disk columnar companion to Arrow's in-memory format; the two are designed to convert efficiently and are maintained by overlapping communities. Polars and DataFusion are engines built directly on Arrow memory, while pandas adopted Arrow-backed dtypes alongside its NumPy legacy. DuckDB has its own internal vectors but imports and exports Arrow without copies. None of these replace Arrow; they are consumers of it.

  • Apache DataFusion

    Extensible Rust query engine built on Apache Arrow offering SQL and DataFrame APIs, Parquet readers and an optimizer for embedding in data systems

    Open sourceOpen source
  • Apache Parquet

    Open columnar file format for analytical data with compression, encodings and statistics, readable by nearly every data engine and library

    Open sourceOpen source
  • DuckDB

    In-process analytical SQL database that queries Parquet, CSV and Arrow data directly from Python, R, Node or the CLI

    Open sourceOpen source
  • pandas

    The standard Python DataFrame library for labeled tabular data, time series and I/O to CSV, Excel, SQL, Parquet and more

    Open sourceOpen source
  • Polars

    Rust-based DataFrame library with Python, Rust, Node and R bindings offering lazy query optimization, multithreading and Arrow-native memory

    Open sourceOpen source
Compare Apache Arrow with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

  • License
    Apache Arrow LICENSE.txt

    https://github.com/apache/arrow/blob/main/LICENSE.txt

    Supports: license_spdx, source_model

    checked Nov 1, 2026
  • Official documentation
    Apache Arrow Documentation

    https://arrow.apache.org/docs

    Supports: capabilities, data_formats

    checked Nov 1, 2026
  • Official documentation
    Arrow Columnar Format specification

    https://arrow.apache.org/docs/format/Columnar.html

    Supports: product_type, capabilities

    checked Nov 1, 2026
  • Official site
    Apache Arrow

    https://arrow.apache.org/

    Supports: name, summary, languages

    checked Nov 1, 2026
  • Repository
    apache/arrow

    https://github.com/apache/arrow

    Supports: repo_url

    checked Nov 1, 2026
Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Delta Lake

Delta Lake project (Linux Foundation; created by Databricks) · Data catalogs & formats

Open table format with a transaction log over Parquet files, created by Databricks and hosted by the Linux Foundation, with Spark, Rust and Python implementations

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

Apache Iceberg

Apache Software Foundation · Data catalogs & formats

Open table format for data lakes that adds ACID snapshots, schema and partition evolution and time travel over Parquet, ORC or Avro files

Apache-2.0Open sourceFreeSelf-hostedNov 2, 2026

MinIO (AIStor)

MinIO, Inc. · Backup & recovery

S3-compatible object storage server; the AGPL community edition was archived in April 2026 and succeeded by MinIO's AIStor Free and AIStor Enterprise editions

MixedFreeEnterpriseSelf-hostedRenamedBroken