Tools · SPBoucher
Menu

Apache Spark

by Apache Software Foundation · Framework · also known as Spark, PySpark, Spark SQL

Distributed data processing engine with DataFrame and SQL APIs in Python, Scala, Java and R, plus structured streaming and MLlib

Apache-2.0 licenseOpen sourceFreeSelf-hostedCloudLocalSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

Apache Spark is a distributed computing engine maintained by the Apache Software Foundation under Apache 2.0. Programs express transformations on DataFrames or SQL tables; the Catalyst optimizer and Tungsten execution layer plan them into stages executed by executors across a cluster managed by Kubernetes, YARN or Spark's standalone manager. Spark SQL reads Parquet, ORC, JSON, CSV, JDBC sources and lakehouse tables (Delta Lake, Iceberg, Hudi); Structured Streaming processes Kafka and file streams with the same API; MLlib and GraphFrames cover machine learning and graph computation. Spark 4 added Spark Connect, a thin client protocol that lets Python, Scala, Go or Rust applications drive a remote cluster, together with ANSI SQL mode by default and a Python data source API.

Spark is the standard engine for large-scale ETL and the foundation of Databricks, Amazon EMR, Google Dataproc and Microsoft Fabric. For data that fits on one machine it is slower to start and heavier to configure than DuckDB or Polars, the JVM and shuffle tuning take expertise, and PySpark user-defined functions pay serialization costs unless written with Arrow-based pandas UDFs.

Capabilities

  • DataFrame and SQL APIs with the Catalyst optimizer
  • Structured Streaming over Kafka and files
  • MLlib machine-learning library
  • Connectors for Parquet, ORC, JSON, JDBC, Delta Lake, Iceberg and Hudi
  • Spark Connect remote client protocol
  • Runs on Kubernetes, YARN or standalone clusters

Typical workflows

  1. Large-scale ETL into a lakehouse
  2. Streaming aggregation of event data from Kafka
  3. Distributed feature engineering and model training

Strengths

  • Scales from a laptop to thousands of nodes
  • Unified batch, streaming and ML APIs
  • Wide managed-service availability

Limitations

  • JVM startup and configuration overhead for small data
  • Shuffle and memory tuning require expertise
  • Python UDFs incur serialization costs

Prerequisites

  • Java runtime; Python for PySpark; a cluster manager for distributed runs

Who it is for

  • Data engineers
  • ML engineers on large datasets

Alternatives & comparison

  • Apache DataFusion

    Extensible Rust query engine built on Apache Arrow offering SQL and DataFrame APIs, Parquet readers and an optimizer for embedding in data systems

    Open sourceOpen source
  • Apache Iceberg

    Open table format for data lakes that adds ACID snapshots, schema and partition evolution and time travel over Parquet, ORC or Avro files

    Open sourceOpen source
  • ClickHouse

    Open-source columnar OLAP database server for fast aggregations over billions of rows, with MergeTree storage, SQL and a managed ClickHouse Cloud

    Open sourceOpen source
  • Dask

    Python library for parallel and distributed computing that scales pandas, NumPy and custom task graphs from a laptop to a cluster

    Open sourceOpen source
  • dbt

    SQL-first transformation framework that compiles templated models, tests and documentation into warehouse tables; open-source Core plus a commercial platform

    MixedOpen source
  • Delta Lake

    Open table format with a transaction log over Parquet files, created by Databricks and hosted by the Linux Foundation, with Spark, Rust and Python implementations

    Open sourceOpen source
  • DuckDB

    In-process analytical SQL database that queries Parquet, CSV and Arrow data directly from Python, R, Node or the CLI

    Open sourceOpen source
  • Polars

    Rust-based DataFrame library with Python, Rust, Node and R bindings offering lazy query optimization, multithreading and Arrow-native memory

    Open sourceOpen source
  • Ray

    Open-source Python framework for distributing tasks, actors and ML workloads across a cluster, with libraries for data processing, training, tuning, serving and reinforcement learning

    Open sourceOpen source
  • Trino

    Distributed SQL query engine that federates queries across data lakes, warehouses and databases through connectors, formerly PrestoSQL

    Open sourceOpen source
Compare Apache Spark with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

  • License
    Spark LICENSE (Apache 2.0)

    https://github.com/apache/spark/blob/master/LICENSE

    Supports: license_spdx, source_model

    checked Nov 1, 2026
  • Official documentation
    Spark documentation

    https://spark.apache.org/docs/latest

    Supports: capabilities, languages, data_formats

    checked Nov 1, 2026
  • Official site
    Apache Spark

    https://spark.apache.org/

    Supports: name, summary

    checked Nov 1, 2026
  • Repository
    apache/spark

    https://github.com/apache/spark

    Supports: repo_url

    checked Nov 1, 2026
Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Apache Arrow

Apache Software Foundation · Data catalogs & formats

In-depth entry

Language-independent columnar in-memory format and libraries that let DuckDB, Polars, pandas, Spark and databases exchange tables without copying

Apache-2.0Open sourceFreeLocalNov 2, 2026

Apache Parquet

Apache Software Foundation · Data catalogs & formats

In-depth entry

Open columnar file format for analytical data with compression, encodings and statistics, readable by nearly every data engine and library

Apache-2.0Open sourceFreeLocalNov 2, 2026

dbt

dbt Labs (Fivetran + dbt Labs) · ETL/ELT & orchestration

In-depth entry

SQL-first transformation framework that compiles templated models, tests and documentation into warehouse tables; open-source Core plus a commercial platform

Apache-2.0Open sourceFreemiumLocalSelf-hostedNov 2, 2026

Elasticsearch

Elastic N.V. · Vector databases

In-depth entry

Distributed search and analytics engine on Apache Lucene with full-text, aggregations, vector search and ES|QL; triple-licensed AGPL, SSPL and Elastic License

AGPL-3.0-only OR SSPL-1.0 OR Elastic-2.0Open sourceFreeSelf-hostedNov 2, 2026

Delta Lake

Delta Lake project (Linux Foundation; created by Databricks) · Data catalogs & formats

Open table format with a transaction log over Parquet files, created by Databricks and hosted by the Linux Foundation, with Spark, Rust and Python implementations

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

Apache DataFusion

Apache Software Foundation · Analytical engines & OLAP

Extensible Rust query engine built on Apache Arrow offering SQL and DataFrame APIs, Parquet readers and an optimizer for embedding in data systems

Apache-2.0Open sourceFreeLocalNov 2, 2026