exo

Open-source distributed inference framework that turns a cluster of consumer devices into a single large model runner. Where ollama runs one model on one machine, exo splits a model across multiple devices — each machine handles a slice of the network's layers, and tokens pass between devices in a ring. The result: you can run models that don't fit in any single machine's memory.

GitHub: exo-explore/exo

How It Works

exo uses pipeline parallelism: the model's layers are distributed across devices. Each device holds a consecutive slice of the full network (e.g., layers 1–40 on machine A, layers 41–80 on machine B for a 80-layer model). During inference:

1. Machine A processes its layers, generates an activation tensor
2. Activation tensor is transmitted to Machine B over the network interface
3. Machine B processes its layers, either sends to the next machine or generates the final token
4. The loop repeats for each generated token

Discovery is automatic via mDNS — devices on the same local network find each other without configuration. The exo process handles partitioning and routing.

The Thunderbolt 5 Angle

Thunderbolt 5 (TB5), shipping with M4 Pro/Max/Ultra Macs starting in late 2024, provides 120 Gbps (15 GB/s) bidirectional bandwidth. That bandwidth is well-matched to pipeline parallelism:

For a 70B model (fp16), the hidden dimension is 8192. Each inter-device activation transfer for a single token is ~16KB. At 100 tokens/sec generation speed, inter-device bandwidth requirement is ~1.6 MB/s — a fraction of TB5's capacity. The bottleneck is compute on each device, not the interconnect.

Where TB5 matters is prefill (processing a long prompt): sending 10,000 tokens at once generates a ~160MB inter-device tensor. At 15 GB/s TB5 bandwidth, that transmits in ~10ms — still fast relative to the compute time for 40 layers on M4 Ultra.

Cable daisy chain: TB5 supports up to 6 devices in a chain. Useful for 3-4 Mac Studios without a Thunderbolt dock.

Practical Configurations

With Llama 4 Maverick (400B total, 17B active MoE): the 17B active parameters mean inference memory is lower than total parameter count suggests. Two Mac Studios with M4 Ultra can run it comfortably.

Backends

• MLX — native Apple Silicon. Fastest on M-series chips. Uses unified memory directly; no GPU memory copies.
• CUDA — for NVIDIA GPUs. Mix and match Mac + NVIDIA in the same cluster.
• tinygrad — lightweight fallback backend. Runs everywhere, lower performance.

exo auto-selects the best available backend per device at runtime.

Supported Models

Llama 3/3.1/3.2/3.3, Mistral, phi-3/4, Qwen 2/2.5, DeepSeek R1/V3, Gemma 2/3, and others. The model registry extends as new models are added. exo downloads model shards from Hugging Face or a local cache; each device only downloads the shard it will run.

Interface

A ChatGPT-compatible API runs on port 52415 by default. Any client that speaks the OpenAI /v1/chat/completions endpoint works with exo — Cursor, Continue, Open WebUI, Aider, etc.

exo also ships a lightweight web UI at the same port for direct conversation.

Comparison to Ollama

For single-device inference, Ollama generally performs better (more mature quantization, wider model support, lower overhead). exo is the right choice when you need more memory than any single device has.

Related

distributed-inference · ollama · lm-studio · claude-code · local-ai-stack

Sources

• exo GitHub
• exo Blog