Claude Code, opencode, aider: every turn re-sends the whole conversation. ARLE keeps the KV cache alive across turns and prefills only what the turn added — turn 20 starts as fast as turn 2.
$ arle serve --backend metal --model-path mlx-community/Qwen3.6-35B-A3B-4bitserving on http://127.0.0.1:8000 · /v1/messages · /v1/chat/completions $ ANTHROPIC_BASE_URL=http://localhost:8000 ANTHROPIC_API_KEY=local claude# turn 1: cold prefill of the system promptprefix-lookup prompt=4850 licensed_blocks=0# turn 2 … turn 12: only the new tokens prefillprefix-attach matched=8640 restored=8640 committed=8651
Most local servers are built around one prompt. An agent session is a loop: the same system prompt and a growing tool history, twenty times over. ARLE is built around the loop.
A coding agent re-sends the whole conversation every turn. ARLE keeps the prior turn's KV on the accelerator, shares prefix pages across requests through a radix cache, and prefills only the tokens the turn added. 12 agent-shaped turns on a MacBook: 180 ms per turn against mlx-lm's 249 ms on the same weights — measured, not projected.
Anthropic /v1/messages with streaming events, tool_use and extended-thinking blocks, and OpenAI /v1/chat/completions with streaming and tools. Claude Code needs one environment variable; any model string is routed to the served model.
Pure Rust from the HTTP layer to the scheduler and the KV cache. Metal runs through an MLX C++ bridge, NVIDIA through FlashMLA, DeepGEMM and DeepEP with CUDA graphs. brew install, one curl line, or a Docker image — clone to first token in minutes.
Every figure on this page resolves to a dated snapshot in docs/experience/wins/ with the command, the environment, and the arm it beat. Speculative decode is verified bit-identical to greedy; restored prefixes are gated against cold prefill with a needle ladder.
One runtime, three surfaces, two backends. Serving, the local agent, and OPD training run the same Rust and model code; dependencies flow strictly downward and infer-core carries no backend dependency. Canonical topology lives in docs/codebase-map.md.
src/main.rs InferenceEngine · LoadedInferenceEngine · backends plug in here Engine<E,K> · scheduler · radix prefix cache BackendExecutor + KvPool seam · ForwardPlan IR — host-only One runnable line per platform, then one environment variable for your agent. Pre-built tarballs and SHAs on each GitHub Release; the curl installer verifies SHA256 before extracting.
$ brew install cklxx/tap/arle $ arle serve --backend metal --model-path mlx-community/Qwen3.6-35B-A3B-4bit
$ curl -fsSL https://github.com/acupof-ai/arle/releases/latest/download/install.sh \ | sh $ arle --doctor
$ docker run --rm --gpus all -p 8000:8000 \
-v $PWD/models:/models:ro ghcr.io/acupof-ai/arle:latest \
serve --backend cuda --model-path /models/Qwen3.6-27B $ ANTHROPIC_BASE_URL=http://localhost:8000 ANTHROPIC_API_KEY=local claude $ export OPENAI_BASE_URL=http://localhost:8000/v1 OPENAI_API_KEY=local # opencode, aider, the openai SDK — anything that speaks the OpenAI API
Dated, reproducible snapshots straight from docs/experience/wins/ and benchmarks/. Decode and prefill are reported separately; nothing is an end-to-end blend.
metal · M4 Pro 48 GB · Qwen3.5-0.8B-MLX-4bit · 12-turn agent conversation, 4.8K-token system prompt, +~350 tokens per turn · same weights and request bytes for both servers
scripts/bench_multiturn_ttft.py --turns 12 --warmup snapshot ↗ metal · M4 Pro 48 GB · Qwen3.6-35B-A3B-4bit (MoE, ~3B active) · 512-in / 128-out · single stream · median of 6
arle serve --backend metal --model-path mlx-community/Qwen3.6-35B-A3B-4bit snapshot ↗ cuda · 1×H20 · Qwen3.6-27B-FP8 + block-drafter speculative decode (DSpark) · 32K-token multi-turn agent prompts · per-request decode
arle serve --backend cuda --spec-type dspark · bench-agent-32k snapshot ↗ cuda · 1×H20 · Qwen3.8-27B-NVFP4 vs Qwen3.6-27B-FP8 · same binary, both arms back to back · nothing resident twice
arle serve --backend cuda --model-path unsloth/Qwen3.8-27B-NVFP4 snapshot ↗ cuda · 4×H20 TP=4 · DeepSeek-V4-Flash · c=1 decode body captured into one CUDA graph per slot · 32K agent prompts
arle serve --backend cuda --tensor-parallel-size 4 snapshot ↗ train · On-Policy Distillation · the teacher is the serving engine, the student trains on its own rollouts · Qwen3.5-4B and Qwen3.5-27B
arle train opd snapshot ↗ Two backends, one runtime contract. Authoritative truth lives in docs/support-matrix.md.
| backend | stability | os / hardware | models | quants | api |
|---|---|---|---|---|---|
cuda | stable | Linux + NVIDIA Ampere+ · 1–8 GPUs (TP / EP) | Qwen3.5 / 3.6 / 3.8 · DeepSeek-V4-Flash · GLM-5.2 | BF16 · FP8 · NVFP4 · W4AFP8 · INT8/FP8 paged KV | Anthropic + OpenAI |
metal | beta | Apple Silicon (M1+) | Qwen3.5 / 3.6 MoE (canonical) · Qwen3 dense · DeepSeek-OCR | MLX 4-bit · BF16 | Anthropic + OpenAI |
cpu | dev only | portable smoke | Qwen3.5 (small) | BF16 | Anthropic + OpenAI |
No queue, no committee — a weekend PR here can move a headline number, and the fronts are public. Start with CONTRIBUTING.md.
Stars are the only metric a small project has. If ARLE made your agent loop faster than what you had, leave one. It decides how much time this gets.
★ Star acupof-ai/arleThe repo at a glance. Everything links back to canonical paths in acupof-ai/arle.