Directory

Distributed data processing engine with DataFrame and SQL APIs in Python, Scala, Java and R, plus structured streaming and MLlib

Python library for parallel and distributed computing that scales pandas, NumPy and custom task graphs from a laptop to a cluster

Open-source software that pools several Apple Silicon Macs (and Linux hosts) into one local inference cluster, sharding large models across devices with MLX

Shell tool that runs commands in parallel over input lines on one or several machines, as a drop-in alternative to xargs and shell loops

Open-source implementation of the MPI message-passing standard used to run tightly coupled parallel programs across the nodes of a cluster

Open-source Python framework for distributing tasks, actors and ML workloads across a cluster, with libraries for data processing, training, tuning, serving and reinforcement learning