Directory

Open-source software that pools several Apple Silicon Macs (and Linux hosts) into one local inference cluster, sharding large models across devices with MLX

Python web framework for building HTTP APIs from type-annotated functions, with automatic validation through Pydantic and generated OpenAPI documentation

Open-source system for scheduling, scaling and operating containerized workloads across a cluster of machines

Open-source Python SDK and self-hosted AI gateway that calls 100+ LLM providers in the OpenAI format, with routing, fallbacks, virtual keys, budgets, spend tracking and logging callbacks

C/C++ inference engine for LLMs and vision-language models with GGUF quantized weights, Metal, CUDA, Vulkan and other backends, a CLI and an OpenAI-compatible server

Apple's open-source array framework for machine learning on Apple Silicon, with NumPy-like Python, C++ and Swift APIs and unified CPU/GPU memory

Python package and CLI for running, serving, quantizing and fine-tuning large language models on Apple Silicon with MLX

Open-source runtime and model manager for running open-weight LLMs locally on macOS, Windows and Linux, with a REST API, OpenAI- and Anthropic-compatible endpoints and optional cloud models

Open-source Python framework for distributing tasks, actors and ML workloads across a cluster, with libraries for data processing, training, tuning, serving and reinforcement learning

High-throughput LLM inference and serving engine with PagedAttention, continuous batching, quantization and an OpenAI-compatible server for GPU and accelerator clusters