gguf-rig

A one-command Podman rig for running local GGUF models on your own GPU.

It builds a custom CUDA llama.cpp from source, then serves every model in your models directory behind a single OpenAI-compatible endpoint, loading them on demand and handing the GPU back when idle. Per-model settings live in one JSON file, so the same image works on a 2060 or a 5090 with whatever models you happen to own.

./rig build && ./rig scan && ./rig up
curl localhost:8099/v1/models

The engine is AbliteratedSuperModelRunner, which needs a specific llama.cpp fork (turboquant KV) rather than upstream. That fork is compiled at image build time from the pin in the app repo, so a clone of this repo plus ./rig build is everything a new machine needs.

Why this exists

Running llama.cpp on a GPU well means getting a dozen details right: the CUDA arch for your card, a fork with the KV cache types your profiles ask for, a context window your VRAM can actually hold, and a way to stop a model hogging the card when you want to play a game. This packages all of that, with the measured reasoning for each choice written down rather than folklore.

GPU access is shared, not exclusive. The container uses CDI, so your desktop, games and other containers keep using the cards. VRAM is only held while a model is loaded.

Requirements

• An NVIDIA GPU of compute capability 7.5 or newer (RTX 20-series onward) and a working driver on the host. Check with nvidia-smi.
• Podman, plus the NVIDIA Container Toolkit for CDI.
• ~15 GB of disk for the image and build, and an hour for the first build. Everything after that is cached.

# Ubuntu / Debian
sudo apt install podman nvidia-container-toolkit

# One-time, and again after every driver update
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk cdi list          # should list nvidia.com/gpu=all

Docker

The image builds and runs under Docker; only the rig wrapper is Podman-specific. Use --gpus all instead of the CDI device (verified working with the nvidia runtime):

docker build -t gguf-rig:latest --build-arg CUDA_ARCHS=120 -f Containerfile .
docker run -d --name gguf-rig --gpus all \
  -p 127.0.0.1:8099:8099 --ipc=host --ulimit memlock=-1 \
  -v /path/to/models:/opt/asmr/models:ro \
  -v "$PWD/config":/config -v "$PWD/logs":/opt/asmr/logs \
  gguf-rig:latest

One real difference: under Podman, rootless user mapping means files written to bind mounts come back owned by you. Under Docker they come back owned by root, including config/models.json, which you then cannot edit without sudo.

Quick start

./rig build                     # compiles llama.cpp with CUDA. Slow, once.

export RIG_MODELS_DIR=/path/to/your/models
./rig scan                      # writes config/models.json, sized to your GPU
./rig up                        # serves on 127.0.0.1:8099
./rig test                      # loads a model and asks for a reply

./rig build detects your card and compiles for exactly it. Nothing to look up. Weights are usually large and already somewhere else, so point at them rather than copying; if you leave RIG_MODELS_DIR unset it uses ./Models.

Do not symlink a GGUF into Models/. The container resolves the link inside its own namespace, where the target does not exist, so the model shows up in listings and then reads as absent. scan warns about this.

If port 8099 is taken (a non-containerized copy of the stack uses it too), set RIG_PORT=8098.

Point any OpenAI-compatible client at http://127.0.0.1:8099/v1:

curl -s localhost:8099/v1/models
curl -s localhost:8099/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"local-qwen-9b","messages":[{"role":"user","content":"hi"}]}'

Setting this up with an AI agent? Point it at AGENTS.md.

Commands

Command	Does
./rig build [--archs N] [--cuda TAG] [--jobs N]	Build the image. Arch is auto-detected
./rig up / down / restart	Lifecycle
./rig shell	Interactive bash inside the running container
./rig status	What is resident, VRAM, queue
./rig models	What /v1/models advertises
./rig scan	Regenerate config/models.json from your models directory
./rig unload	Free the GPU now, keep serving
./rig logs -f	Follow logs
./rig test	End-to-end check against the running server

GPU access: shared, not passthrough

--device nvidia.com/gpu=all is CDI. It bind-mounts the driver libraries and /dev/nvidia* into the container. It is not VFIO passthrough: the host keeps full use of the cards, your desktop and games are unaffected, and other containers can use them too. VRAM is only occupied while a model is actually loaded, and llmd unloads after LLMD_IDLE_SECONDS (default 900).

If the GPU is not visible inside the container, regenerate the CDI spec on the host. This is needed after every driver update:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk cdi list

Per-model configuration

config/models.json is the whole model layer. It is a bind mount, so editing it needs no rebuild, only ./rig restart.

{
  "defaults": {
    "args": ["-ngl", "99", "-fa", "on", "--jinja", "--load-mode", "none"]
  },
  "models": [
    {
      "name": "qwen-9b-128k",
      "gguf": "Qwen3.5-9B-abliterated-Q5_K_M.gguf",
      "aliases": ["fast", "day"],
      "context": 131072,
      "args": ["-ctk", "f16", "-ctv", "f16"],
      "thinks": true,
      "note": "9B at 131k, f16 KV"
    },
    {
      "name": "qwen-9b-256k",
      "gguf": "Qwen3.5-9B-abliterated-Q5_K_M.gguf",
      "aliases": ["long"],
      "context": 262144,
      "args": ["-ctk", "turbo4", "-ctv", "turbo3"]
    }
  ]
}

Field	Meaning
name	Advertised as local-<name>
gguf	Filename in ./Models
aliases	Extra names clients may send. Must be unique across models
context	Window per slot. Defaults to the GGUF's declared context
args	Appended to defaults.args, passed to llama-server
thinks	Usually omit. Detected from the chat template
kv_kib_per_token	Usually omit. Computed from GGUF metadata

The same GGUF can appear repeatedly at different windows or cache types, which is how the upstream profile table works. defaults.args come first and a model's own args can override them, since llama-server honours the last occurrence of a repeated flag.

Thinking models

Reasoning is on by default wherever the model supports it, because that is where a reasoning model's quality comes from. Turning it off would quietly give you a worse model.

You do not have to declare it. scan reads tokenizer.chat_template from the GGUF and looks for enable_thinking, <think> or reasoning_content; omitting thinks from a hand-written config triggers the same detection. An explicit true or false always wins.

The one trap is the token budget. A thinking model writes into reasoning_content before it produces any content, so a request with a small max_tokens spends the whole budget reasoning and returns an empty string with finish_reason: "length". That reads as a broken server.

llmd handles this: below a 2000-token floor (LLMD_THINKING_FLOOR) it disables thinking for that one request, so a client asking for a short answer gets a short answer rather than nothing. Above the floor, reasoning is kept. Verified both ways: at max_tokens: 4096 the model reasoned for 1382 characters then answered, and at max_tokens: 200 it answered directly.

To force it off for a single request, send it yourself:

{"chat_template_kwargs": {"enable_thinking": false}}

Why kv_kib_per_token is derived, not guessed

llmd divides the VRAM budget by this number to decide how many concurrent slots fit. Too low and it promises more concurrency than the card can hold; that does not fail loudly, it silently pages through CUDA unified memory and reads as "the model got slow for no reason".

So it is computed from the GGUF's own metadata with the same formula as upstream's bin/kv-calc, including the hybrid-attention case where only one layer in full_attention_interval pays per-token KV. Verified against kv-calc on a real model: both report 32.00 KiB/token for the 9B at f16.

Pushing the window

There is no artificial context ceiling. scan picks the largest power-of-two window your card can actually pay for, bounded only by the model's own trained context, and you can raise it further by hand. 256k is verified working here on a 12 GB card; 512k and 1M are reachable with enough VRAM and the right cache type.

Three things decide how far you get.

1. The KV cache has to fit. Cost per token comes from the model's architecture, and scan reads it from the GGUF. A hybrid-attention model is dramatically cheaper than a dense one, because only the periodic full-attention layers pay per token:

podman exec gguf-rig python3 bin/kv-calc models/your-model.gguf

2. Quantise the cache for large windows. This is what the turboquant fork exists for. f16 at 262144 on a 9B is ~8 GiB of cache on top of the weights, which does not fit in 12 GB; turbo4/turbo3 does:

{ "name": "9b-256k", "gguf": "Qwen3.5-9B.gguf", "context": 262144,
  "args": ["-ctk", "turbo4", "-ctv", "turbo3"] }

Worth knowing: f16 is faster wherever it fits. Measured on the 9B, f16 and turbo are within 2% on an empty cache, but turbo costs ~40% once the window is actually full. Use turbo to reach a size f16 cannot, not as a default.

3. Above the model's declared context you need rope scaling. llama-server silently clamps every slot to n_ctx_train, so asking for more is not enough on its own: slots stay small and long prompts are rejected with "exceeds the available context size". llmd patches that metadata field for you via gguf-set-ctx, but that rewrites the GGUF in place, so the models directory must be writable:

RIG_MODELS_RO=0 ./rig up

Add the rope arguments in the profile, e.g. "args": ["--rope-scaling", "yarn", "--rope-scale", "2"].

Quality is the real limit, not loading. Retrieval on this stack held at 431k tokens up to mid-depth under YaRN 2x, but failed at 90% depth. A 1M window loads and runs; treat it as a ceiling for bulk ingest rather than a place to hide important facts.

Splitting across GPUs and CPU

Any llama-server flag works in args, so the usual placement tricks apply:

"args": ["-ts", "9,3"]          // split weights across two GPUs, 9:3 ratio
"args": ["-ot", "exps=CPU"]     // MoE experts in system RAM, attention on GPU
"args": ["-ngl", "20"]          // only 20 layers on the GPU, rest on CPU

-ot exps=CPU is how a 30B MoE runs on a 12 GB card at all. -ngl below the layer count is the general fallback when weights simply exceed VRAM; it works, and it is much slower, so it is never a generated default.

Two traps, both measured:

• Never use --no-kv-offload. Putting the KV cache in system RAM measured 1.03 tok/s against 91.6 tok/s with it on the GPU, roughly 90x. Overflow is handled by CUDA unified memory instead, which pages on demand rather than routing every token over PCIe.
• A second GPU is not free. Whichever card holds a tensor runs that tensor's math, so a slow card on a narrow PCIe link drags generation down. On this machine, letting a 2060 on a gen1 x4 link participate cost ~40%. Some older cards also lack kernels for newer quant types and abort at load.

Other GPUs

./rig build reads your cards from nvidia-smi and compiles for exactly those, so normally there is nothing to choose:

./rig build                      # detects, e.g. "archs 120" on a 5070 Ti

Override when you are building for a machine other than this one, or want to trim the image:

./rig build --archs 120          # every RTX 50-series card
./rig build --archs "89;120"     # a mixed 40 + 50 fleet

Card	Arch
RTX 2050 - 2080 Ti (Turing)	75
RTX 3050 - 3090 Ti (Ampere)	86
RTX 4050 - 4090 (Ada)	89
RTX 5050, 5060, 5060 Ti, 5070, 5070 Ti, 5080, 5080 Super, 5090	120

The entire RTX 50-series is one arch. Every GeForce Blackwell part, from the 5050 to the 5090 and including every Ti and Super variant, is compute capability 12.0, so --archs 120 covers all of them and a build for one runs on any other. (sm_121 exists but is DGX Spark, not a GeForce card.)

RTX PRO Blackwell workstation cards are also 12.0, so they work with the same 120 build.

Two constraints worth knowing:

• 50-series needs CUDA >= 12.8. The default toolkit here is 13.2.1, so this only matters if you pin an older --cuda by hand; rig refuses that combination rather than letting nvcc fail an hour in. Note that some reports find CUDA 13.1 specifically slow or crash-prone on sm_120, which is why the default is 13.2.1.
• CUDA 13 dropped everything below compute 7.5. Maxwell, Pascal and Volta cannot be targeted at all and need an older toolkit:

./rig build --cuda 12.9.1-devel-ubuntu24.04 --archs 61

Making changes inside the container

./rig shell

Lands in /opt/asmr as root, with the full source tree and toolchain. The app is a git checkout, so you can edit, rebuild and test in place:

vi bin/llmd
cmake --build src/llama.cpp/build -j"$(nproc)"   # after a llama.cpp edit

Changes inside the container are lost when it is removed, since the code lives in the image, not on a mount. That is deliberate: the image is reproducible. Make a change permanent by pushing it upstream and running ./rig build, or by bind-mounting your own checkout over /opt/asmr while developing.

./rig up also runs llama-server and every upstream tool:

podman exec -it gguf-rig llama-server --help | grep turbo
podman exec -it gguf-rig python3 bin/kv-calc models/your-model.gguf

Layout

Containerfile                 image: CUDA base + llama.cpp fork build
rig                           the wrapper you actually use
AGENTS.md                     setup procedure for an AI agent
compose.yaml                  for existing compose stacks
scripts/entrypoint.sh         GPU check, config generation, launches llmd
scripts/profiles_overlay.py   replaces llmd's hardcoded PROFILES with JSON
config/models.json            per-model configuration (generated by scan)
config/models.example.json    a tuned reference config
Models/                       default models dir; see RIG_MODELS_DIR
logs/  kv-slots/              llmd logs and KV snapshots

Build notes and traps

These two cost an hour each to discover. Both are container-specific and neither happens when building on a normal machine with a driver installed.

undefined reference to cuMemCreate at the final link. A *-devel image ships the CUDA toolkit but deliberately has no driver: the real libcuda.so.1 belongs to the host and is injected at run time by CDI. ggml-cuda calls the driver API and links it PRIVATE, which satisfies libggml-cuda.so itself but does not propagate, so linking llama-server against it fails with a dozen unresolved cuMem*/cuDevice* symbols. This is llama.cpp#23357, open and not fork-specific. The fix is to link the stub explicitly:

-DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda"
-DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda"

Safe because the stub's SONAME is libcuda.so.1, so the recorded NEEDED entry is the real driver's name. What must never happen is the stubs directory reaching the runtime search path: the process would then bind the do-nothing stub instead of the injected driver and every CUDA call would fail. The build asserts against that with readelf.

llama-server --help cannot run at build time. The verification step originally just ran --help | grep turbo2 and failed, which reads as "the build has no turbo KV" and sends you hunting the wrong fork. The binary was fine; the dynamic loader could not resolve libcuda.so.1 and killed the process before main(). LD_LIBRARY_PATH=$CUDA_STUBS for that one command fixes it, and is never set at runtime.

Verification is a separate layer from the compile. The compile is ~50 minutes of nvcc. When an assertion lived in the same RUN, a failed check discarded the whole layer and recompiled from scratch. Split, a failed check costs seconds.

BUILD_JOBS defaults to three quarters of the cores via ./rig build, not all of them. A full-core build pins every thread for most of an hour on a machine that is also a desktop, and the wall-clock saving is small because the build is bandwidth-bound before it is thread-bound. Override with --jobs N.

Design notes and traps

Why the profiles are overlaid rather than forked. Upstream bin/llmd has a PROFILES tuple hardcoded to four GGUF filenames and to a specific 4070+2060 box. In a shared image that means /v1/models advertising four files most people do not have. scripts/profiles_overlay.py is imported before llmd.main() and rebuilds llmd.PROFILES from JSON. Nothing is patched, since PROFILES is read at request time by resolve() and /v1/models.

bin/llmd has no .py suffix, so it cannot be imported normally. The entrypoint loads it with SourceFileLoader, which also means main() is not invoked at import and the overlay can be applied first.

Models is mounted read-only by default. Loading a profile can call gguf-set-ctx, which rewrites the GGUF's declared n_ctx_train in place via a writable mmap, because llama-server otherwise silently clamps every slot to that value. Useful upstream, surprising when it edits weights you copied in. Set RIG_MODELS_RO=0 ./rig up to allow it; contexts above the model's declared value will not otherwise take effect.

The build verifies turbo KV and fails if absent. Upstream ggml-org/llama.cpp has no turbo2/3/4 cache types, and a build without them gets all the way to serve time before dying on an unknown -ctk. The image checks llama-server --help for turbo2 at build time instead.

The llama.cpp pin is read from the app repo, not duplicated here. The Containerfile greps src/LLAMA-CPP-PIN.md for the remote, tag and branch, and fails loudly if any comes back empty. Bumping the pin upstream needs no edit here.

--restart unless-stopped plus a 30s stop timeout. llmd handles SIGTERM so it can stop llama-server and release VRAM; killing it faster orphans a process holding the GPU.

Never publish this on 0.0.0.0 casually. There is no authentication. RIG_BIND exists but loopback is the default for a reason.

Verified on

Built and run end to end on 2026-09-13, Ubuntu 26.04, Podman 5.7.0 rootless, driver 595.84, RTX 4070 Super + RTX 2060, --archs 75;89:

Check	Result
Image build	8.83 GB, ~50 min at 9/12 jobs
turbo2/3/4 KV types	present (asserted during the build)
GPUs inside the container	both visible via CDI
Arch auto-detection	reported 75;89, matching a hand-built image bit for bit
9B Q5_K_M at 131k (f16 KV)	11.3 GiB VRAM, replied over /v1/chat/completions
9B Q5_K_M at 262k (turbo KV)	n_ctx_slot = 262144, replied correctly
KV auto-derivation	32.00 KiB/token, matches bin/kv-calc exactly
VRAM-aware scan	65,536 for the 9B, warned that a 12.6 GiB model exceeds 12 GiB
./rig unload	VRAM back to 1.4 GiB (desktop only)
./rig down	container removed, no orphaned llama-server
Docker --gpus all	both GPUs visible (build not run end to end)
Host impact	desktop and a host llmd on :8099 unaffected throughout
Public clone	anonymous clone rebuilt to an identical image hash

Environment variables

Variable	Default	Effect
RIG_PORT	8099	Host port
RIG_BIND	127.0.0.1	Host interface
RIG_MODELS_DIR	./Models	Where the GGUFs really are
RIG_GPUS	all	CDI selector, e.g. 0 for one card
RIG_MODELS_RO	1	Mount ./Models read-only
LLMD_IDLE_SECONDS	900	Unload after this much idle time
LLMD_CUDA_DEVICES	0	Which visible GPU serves
LLMD_MAX_CONCURRENT	4	Slot ceiling, VRAM permitting
LLMD_DESKTOP_RESERVE_MIB	1500	VRAM left for the desktop