Directory

Distributed data processing engine with DataFrame and SQL APIs in Python, Scala, Java and R, plus structured streaming and MLlib

Python library for parallel and distributed computing that scales pandas, NumPy and custom task graphs from a laptop to a cluster

Open-source software that pools several Apple Silicon Macs (and Linux hosts) into one local inference cluster, sharding large models across devices with MLX

Shell tool that runs commands in parallel over input lines on one or several machines, as a drop-in alternative to xargs and shell loops

Open-source system for scheduling, scaling and operating containerized workloads across a cluster of machines

Open-source implementation of the MPI message-passing standard used to run tightly coupled parallel programs across the nodes of a cluster

Open-source Python framework for distributing tasks, actors and ML workloads across a cluster, with libraries for data processing, training, tuning, serving and reinforcement learning

Open-source cluster resource manager and job scheduler for Linux clusters, allocating nodes, launching jobs and managing queues on many HPC systems

High-throughput LLM inference and serving engine with PagedAttention, continuous batching, quantization and an OpenAI-compatible server for GPU and accelerator clusters