Tools · SPBoucher
Menu

GNU Parallel

by GNU Project (Ole Tange) · CLI tool · also known as parallel

Shell tool that runs commands in parallel over input lines on one or several machines, as a drop-in alternative to xargs and shell loops

GPL-3.0-or-later licenseOpen sourceFreeLocalSelf-hostedSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

GNU Parallel is a command-line tool from the GNU Project, written and maintained by Ole Tange, for executing jobs in parallel on one or more computers. It reads a list of inputs (files, hosts, URLs, table names or lines from a pipe) and runs a command once per input, keeping a configurable number of jobs running at a time. Output is buffered so that results appear exactly as if the commands had run sequentially, which lets it slot into pipelines. The options deliberately mirror xargs, and it can replace most for loops in shell scripts.

With --sshlogin it distributes jobs to remote hosts over SSH, transferring input files and returning results, which makes it a lightweight way to use a handful of machines, such as a cluster of Macs, without a scheduler. Other features include job logs, retries, resumable runs, progress display and splitting large inputs into chunks for parallel processing.

It is licensed under the GPL version 3 or later and is available through Homebrew and all major Linux distributions. The program asks users to cite it in academic work (silenceable with --citation), which some find intrusive, and it is a Perl script that depends on Perl being present; it does not provide scheduling, queues or fault tolerance beyond simple retries.

Capabilities

  • Parallel execution of a command over input lines
  • Remote execution over SSH with file transfer
  • Ordered output buffering for pipeline use
  • Job logs, retries and resumable runs
  • Splitting large inputs into blocks

Typical workflows

  1. Convert thousands of files using all cores
  2. Run a script on each host in a list via SSH
  3. Chunk a large text file through a processing command

Strengths

  • Zero setup; one Perl script
  • Familiar xargs-style options
  • Works across machines without a scheduler

Limitations

  • No queueing, priorities or persistent state
  • Citation notice must be acknowledged
  • Depends on Perl and SSH for remote jobs

Prerequisites

  • Perl
  • SSH access for remote hosts

Who it is for

  • Shell users processing many files
  • Researchers running embarrassingly parallel jobs

Alternatives & comparison

  • Dask

    Python library for parallel and distributed computing that scales pandas, NumPy and custom task graphs from a laptop to a cluster

    Open sourceOpen source
  • Ray

    Open-source Python framework for distributing tasks, actors and ML workloads across a cluster, with libraries for data processing, training, tuning, serving and reinforcement learning

    Open sourceOpen source
  • Slurm Workload Manager

    Open-source cluster resource manager and job scheduler for Linux clusters, allocating nodes, launching jobs and managing queues on many HPC systems

    Open sourceOpen source
Compare GNU Parallel with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

  • Official documentation
    GNU Parallel manual

    https://www.gnu.org/software/parallel/man.html

    Supports: capabilities, license_spdx, limitations

    Manual states GPL version 3 or later

    checked Nov 1, 2026
  • Official site
    GNU Parallel

    https://www.gnu.org/software/parallel

    Supports: name, summary, description

    checked Nov 1, 2026
  • Repository
    GNU Parallel on Savannah

    https://savannah.gnu.org/projects/parallel

    Supports: repo_url

    checked Nov 1, 2026
Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Kubernetes

Cloud Native Computing Foundation · Containers & virtualization

In-depth entry

Open-source system for scheduling, scaling and operating containerized workloads across a cluster of machines

Apache-2.0Open sourceFreeSelf-hostedNov 2, 2026

vLLM

vLLM project (originated at UC Berkeley Sky Computing Lab) · Local LLM inference & serving

In-depth entry

High-throughput LLM inference and serving engine with PagedAttention, continuous batching, quantization and an OpenAI-compatible server for GPU and accelerator clusters

Apache-2.0Open sourceFreeSelf-hostedNov 2, 2026

Apache Spark

Apache Software Foundation · Analytical engines & OLAP

Distributed data processing engine with DataFrame and SQL APIs in Python, Scala, Java and R, plus structured streaming and MLlib

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

Open MPI

Open MPI consortium · Distributed & parallel computing

Open-source implementation of the MPI message-passing standard used to run tightly coupled parallel programs across the nodes of a cluster

BSD-3-Clause-Open-MPIOpen sourceFreeLocalSelf-hostedBroken

exo

Exo Labs · Distributed & parallel computing

Open-source software that pools several Apple Silicon Macs (and Linux hosts) into one local inference cluster, sharding large models across devices with MLX

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026