Distributed Inference

Running a single model inference request across multiple devices. The model's weights are split across machines; each machine handles its slice and passes intermediate results to the next. The goal is to run models that don't fit in any single device's memory, or to parallelize computation across a cluster for lower latency.

Distinct from distributed training (which parallelizes gradient computation across many examples) — distributed inference runs a single request across a cluster in real time.

Why It's Needed

Model weights grow faster than single-device memory. A few reference points:

• Llama 3.1 70B at fp16: 140 GB — doesn't fit on a single A100 (80GB), fits on two
• Llama 4 Maverick (400B total): ~800 GB fp16 — requires 10× A100s, or ~4× Mac Studios with M4 Ultra at int4
• GPT-4 class models (estimated 1T+ parameters): requires datacenters

For consumer hardware, the practical limit without distributed inference is roughly 70B at fp16 on the largest Mac Studio (192GB unified memory), or 405B at int4 quantization on that same machine. Two machines doubles the ceiling.

Parallelism Strategies

Pipeline parallelism: The model's layers are split sequentially across devices. Device 1 holds layers 1–40, device 2 holds layers 41–80. Inference passes through device 1 first, then device 2. Each device is mostly idle while the other works — GPU utilization is lower than tensor parallelism, but inter-device communication per token is minimal (just the activation tensor at the boundary). This is what exo uses.

Tensor parallelism: Individual layers are split across devices. For a 4-head attention layer, each device computes 1 head's full computation in parallel. All devices must communicate at every layer, requiring very high bandwidth. Used in production datacenters with NVLink or InfiniBand; latency is prohibitive over standard Ethernet.

Sequence parallelism: Long sequences are split across devices in the context dimension. Device 1 processes tokens 1–500K, device 2 processes tokens 500K–1M. Useful for million-token context on memory-constrained hardware. Emerging technique; less mature.

Speculative decoding: One small "draft" model generates candidate tokens fast; a large "verifier" model confirms in parallel. When correct, you get the quality of the large model at the speed of the small model. Can run across devices but is primarily a latency optimization within a single device.

The Bandwidth Bottleneck

Distributed inference is gated by the bandwidth between devices. The inter-device transfer per token per layer boundary:

hidden_dim × dtype_bytes = bytes per transfer

For Llama 3.1 70B (hidden dim 8192, fp16): 8192 × 2 = 16 KB per token per layer boundary.

At 100 tokens/sec: 1.6 MB/s — easily handled by Gigabit Ethernet (125 MB/s), let alone Thunderbolt 5 (15 GB/s) or NVLink (900 GB/s).

The bottleneck in practice is prefill — processing a 10K token prompt generates a single 160 MB transfer at the layer boundary. That's why bandwidth tier matters more for long-context workloads than for short generation tasks.

Apple Silicon Architecture

Apple Silicon has a key advantage for distributed inference: unified memory. CPU, GPU, and Neural Engine share the same memory pool, meaning the model weights are accessible from all compute units without PCIe bus transfers. This is why a Mac Studio M4 Ultra with 192GB can run a model that wouldn't fit in an NVIDIA A100's 80GB VRAM — the entire 192GB is equally fast for the GPU to read.

In a distributed exo cluster across multiple Macs:

• Each device loads its layer slice into unified memory
• MLX (Apple's ML framework) runs the layers natively on the M-series GPU
• Thunderbolt transfers carry activations between machines
• No PCIe bottleneck on either side of the transfer

This makes Apple Silicon uniquely suited to Thunderbolt-based pipeline parallelism. NVIDIA multi-GPU setups require explicit CUDA memory management and NVLink or PCIe for inter-GPU communication.

Production vs Consumer Distributed Inference

Production distributed inference (vLLM, TensorRT-LLM, DeepSpeed, Ray) targets datacenter environments:

• High-bandwidth interconnects (NVLink, InfiniBand)
• GPU compute rather than unified memory
• Advanced optimizations: KV cache reuse, paged attention, continuous batching
• Handles concurrent users (thousands of requests simultaneously)

Consumer distributed inference (exo, llamafile, llama.cpp distributed):

• Consumer interconnects (Thunderbolt, Ethernet, Wi-Fi)
• Unified memory (Apple) or standard VRAM (NVIDIA)
• Single-user or small-group usage
• Focus on running models that exceed single-device memory

The gap is closing. exo's MLX backend on Apple Silicon achieves 15–30 tokens/sec on a 70B model on a single M4 Ultra, comparable to early vLLM performance on A100s.

Related

exo · ollama · agentic-workflows · gpu-clouds · extended-thinking

Sources

• exo GitHub
• Apple MLX Documentation
• vLLM Documentation