Status: pre-alpha, module landed
Runs in production today
The lane runs in production on a shared single-GPU cluster in its original form — continuous batching, fit-aware MoE expert-offload, and the store-as-registry config generator, all live.
serving module landed
The generalized serving module
(nixidyModules.serving) is extracted and published in
this repo — and dogfooded back: the originating cluster runs this
generalized module in production since 2026-07-22. It packages the full lane: llama-swap + llama.cpp
broker, LiteLLM front door, store-scan config generator, and the GPU
contract surface (priority class, device token, Recreate strategy) as
options.
No invented benchmarks
Nothing on this page claims a module, benchmark, or behavior that isn't real. This lane implements behaviors B4/B10/B14/B15 of the nixgpu contract — the honest spec it's built and tested against.
Native runtime offload, proven under contention
The MoE fit-aware expert-offload was proven live today with the card genuinely contended — a real chat completion served successfully with only about 2.5 GiB free on a 16 GiB card — rather than assumed from a static VRAM budget; this is same-day validation under adversarial load, not a claim of long-term soak.
The pitch
The store IS the registry
Serving mode comes from the subdirectory (embeddings/
→ embedder, rerankers/ → reranker, anything
else → chat); context length and chat template come from GGUF
metadata; the rest is sane defaults. Adding a model = dropping a
file.
Fit-aware, not naive
An oversized dense model is skipped — never the card-reset an oversized full-offload load risks. An oversized Mixture-of-Experts model is served anyway, experts on CPU RAM, attention + KV on the GPU. Shard sets collapse to their first member; stable app-facing names come from a one-line alias file beside the model.
One broker, no VRAM hogs
Apps never start their own GPU model server — they point at the one front door with a key and a model name, and request no GPU of their own. Models load on demand in seconds (a warm page cache does the heavy lifting) and idle models unload.
Concurrency is first-class, twice
Same-model requests interleave via continuous batching; several small models co-reside and serve in parallel when they fit together.
Module
serving | The full lane as a nixidy module: llama-swap + llama.cpp broker, LiteLLM front door, store-scan config generator, GPU contract surface as options. Landed and dogfooded — the originating production cluster runs this generalized module. See modules/serving/README.md for the option table. |
Requirements (deliberate, not negotiable)
| Delivery path | Declarative GitOps only — nixidy-rendered manifests synced by Argo CD, the spine shipped by the sibling nixk3s project. Hand-applied YAML is not supported. |
| GPU scheduling | Priority-based scheduling and VRAM reclaim come from the sibling nixgpu project; this lane declares its three-line contract and never manages the card itself. |
| Scope | Serving lane only — one shared LLM broker. Other tenants (image generation, TTS, …) ship as app modules in the sibling nixapps project. |
Related projects
nixgpu
Priority-based sharing of one GPU — the substrate this lane runs on. See nixgpu.corbet.ch.
nixk3s
Bare-metal k3s on NixOS + the nixidy → Argo CD GitOps spine this lane deploys onto. See nixk3s.corbet.ch.
nixapps
Curated tenant app modules (image generation, TTS, …) that consume the same contracts — and where this lane started before it graduated into its own project. See nixapps.corbet.ch.