Apache Parquet

Name: Apache Parquet
Author: Apache Software Foundation

In-depth entry

by Apache Software Foundation · Specification · also known as Parquet

Open columnar file format for analytical data with compression, encodings and statistics, readable by nearly every data engine and library

Apache-2.0 licenseOpen sourceFreeLocalSelf-hostable

Visit official site Compare

Link verified Nov 2, 2026

Overview

Apache Parquet is a columnar storage file format specified by an Apache Software Foundation project and implemented in C++ (via Apache Arrow), Java (parquet-java), Rust (via arrow-rs), Go and other languages. A Parquet file organizes rows into row groups and, within each, stores every column contiguously as pages with per-column encodings (dictionary, run-length, delta, byte-stream split) and compression codecs (Snappy, Zstandard, gzip, LZ4, Brotli). The footer carries the schema plus min/max statistics, and optional page indexes and Bloom filters let readers skip row groups and pages that cannot match a predicate. Nested data is represented with the Dremel record-shredding model, so lists and structs are first-class.

Parquet is the default storage layer for data lakes and lakehouses: Spark, Trino, DuckDB, Polars, pandas (through PyArrow or fastparquet), DataFusion, ClickHouse, BigQuery, Snowflake and Athena all read and write it, and table formats such as Apache Iceberg and Delta Lake manage collections of Parquet files. For a research group it is the natural archive for tick or minute-bar histories, filings extractions and panel datasets: columns compress well, partitioned directories allow year/month pruning, and a single laptop can query hundreds of gigabytes directly.

The format is write-once: files are immutable and appends or updates mean rewriting files or using a table format on top. Small files and tiny row groups hurt performance, schema evolution needs care across engines, and features such as the newer encodings, page indexes or geospatial logical types are not supported uniformly by every reader, so pinning writer settings for compatibility is common practice. The specification is Apache 2.0.

Capabilities

Columnar layout with row groups, column chunks and pages
Encodings: dictionary, RLE/bit-packing, delta, byte-stream split
Compression codecs including Snappy, Zstandard, gzip and LZ4
Column statistics, page indexes and Bloom filters for predicate pushdown
Nested types via the Dremel shredding model
Logical types for timestamps, decimals, UUIDs and more
Implementations in C++, Java, Rust, Go, Python and others

Typical workflows

Archive minute-bar or tick data partitioned by year and symbol
Store ETL outputs for DuckDB, Polars or Spark consumers
Exchange datasets between Python, R and JVM tooling
Underlying files for Iceberg or Delta Lake tables

Strengths

Compact storage and fast column scans
Supported by virtually all analytical engines
Statistics enable data skipping
Vendor-neutral Apache 2.0 specification

Limitations

Immutable files; updates require rewrites or a table format
Many small files degrade performance
Feature support varies between implementations
Not suited to row-by-row transactional access

Prerequisites

A library or engine with a Parquet reader (PyArrow, DuckDB, Polars, Spark…)

Who it is for

Data engineers
Quantitative and empirical researchers
Analysts working on data lakes

Alternatives & comparison

Apache Arrow is the in-memory counterpart designed for fast conversion to and from Parquet; Arrow IPC files can also be used on disk but with less compression. Iceberg and Delta Lake are table formats layered over Parquet files that add transactions, schema evolution and time travel. DuckDB and SQLite are databases with their own file formats; DuckDB reads Parquet natively and is often used as the query layer over Parquet archives.

Apache Arrow
Language-independent columnar in-memory format and libraries that let DuckDB, Polars, pandas, Spark and databases exchange tables without copying
Open sourceOpen source
Apache Iceberg
Open table format for data lakes that adds ACID snapshots, schema and partition evolution and time travel over Parquet, ORC or Avro files
Open sourceOpen source
Delta Lake
Open table format with a transaction log over Parquet files, created by Databricks and hosted by the Linux Foundation, with Spark, Rust and Python implementations
Open sourceOpen source
DuckDB
In-process analytical SQL database that queries Parquet, CSV and Arrow data directly from Python, R, Node or the CLI
Open sourceOpen source
SQLite
Public-domain embedded SQL database engine that stores a whole database in one file and ships inside most operating systems and browsers
Open sourceOpen source

Compare Apache Parquet with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

License
parquet-format LICENSE (Apache 2.0)
https://github.com/apache/parquet-format/blob/master/LICENSE
Supports: license_spdx, source_model
checked Nov 1, 2026
Official documentation
Parquet documentation
https://parquet.apache.org/docs
Supports: capabilities
checked Nov 1, 2026
Official documentation
Parquet file format specification
https://parquet.apache.org/docs/file-format
Supports: capabilities, product_type
checked Nov 1, 2026
Official site
Apache Parquet
https://parquet.apache.org/
Supports: name, summary
checked Nov 1, 2026
Repository
apache/parquet-format
https://github.com/apache/parquet-format
Supports: repo_url
checked Nov 1, 2026

Confidence: High
Reviewed: Nov 1, 2026
Links: Verified · Nov 2, 2026

Spotted an error?

MinIO (AIStor)

MinIO, Inc. · Backup & recovery

S3-compatible object storage server; the AGPL community edition was archived in April 2026 and succeeded by MinIO's AIStor Free and AIStor Enterprise editions

MixedFreeEnterpriseSelf-hostedRenamedBroken