AbliteratedSuperModelRunner

A measured, two-profile local LLM stack for **12GB RTX 4070 Super + 6GB RTX 2060

• 64GB RAM + NVMe**, built on llama.cpp. Everything lives on the NVMe.

Design goal: a 500k-1M token window, at usable speed, on abliterated (refusal-removed) coder models, without permanently monopolising the GPU.

Every number below was measured on this box, not estimated. Re-run bin/bench-stack to reproduce.

Install / restore

bin/install              # idempotent: build, fetch models, install services
bin/install --no-models  # skip the model download

A clone is deliberately not self-contained: the llama.cpp fork (~217 MB) and the four GGUFs (~36 GB) are not in git. bin/install rebuilds the fork from the pin in src/LLAMA-CPP-PIN.md plus the patches beside it, and bin/quantize-models rebuilds the weights (see QUANTIZING.md).

Platform. Linux is the primary target and the only one the numbers below were measured on. Windows works too, via Git Bash / MSYS2 / WSL for the shell scripts; bin/llmd itself is plain Python, run it as python bin\llmd since it has no .py suffix to associate. Where systemd is absent, bin/install writes bin/llmd-start (and a .cmd that does exactly that) instead of units. The per-OS mechanisms and the traps involved are tabulated in AGENTS.md under "Running on Windows". Verify a checkout with:

python3 tests/test_os_agnostic.py   # 12 checks, both OS branches
bash    tests/test_gitignore.sh     # 33 checks, artifacts vs source

Quick start

bin/llm day       # 9B, 524k window, 4070 does the work
bin/llm night     # 30B coder, 327k window
bin/llm cook      # night profile + overnight job queue
bin/llm off       # stop everything, hand the GPUs back
bin/llm status    # what is running, live window, VRAM, snapshots

OpenAI-compatible endpoints:

Profile	URL	Model name
day	http://127.0.0.1:8090/v1	ornith-9b
night	http://127.0.0.1:8091/v1	qwen-coder-30b-ablit

(Ports are 8090/8091 because 8080 was already taken on this machine.)

Models (both abliterated, as requested)

Role	Model	Quant	Size
Day	Ornith-1.5-9B-heretic	i1-IQ3_M	4.2 GB
Night	Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated	i1-IQ3_M	13 GB

Verified working: the 9B emits clean tool-call JSON, so agent loops are intact after abliteration.

The 9B is a thinking model. It writes into reasoning_content before content. With a small max_tokens you get an empty reply and it looks broken. Either allow ~500+ tokens, or disable thinking:

{"chat_template_kwargs": {"enable_thinking": false}}

Measured results

The finding that matters most

Grok's plan put the KV cache in system RAM (--no-kv-offload) to protect game VRAM. That is catastrophic for this workload:

KV cache location	Generation speed
System RAM (--no-kv-offload)	1.03 tok/s
On the GPU	91.6 tok/s

~90x. At 431k tokens the RAM-cache config generated at 1.03 tok/s: a 40-token answer took 38 seconds after a 13-minute prefill. This stack never uses --no-kv-offload. Overflow is handled by CUDA unified memory instead, which pages on demand rather than routing every token over PCIe.

Windows (day profile, 4070 compute + KV, 2060 holding weights)

Preset	Window	Rope	Generation	Prefill
fast	262,144	native	32-91 tok/s	~2600-3200 tok/s
long (default)	524,288	YaRN 2x	62 tok/s	~1000-3100 tok/s
max	1,048,576	YaRN 4x	62 tok/s	degraded quality

Your 40+ tok/s target is met at 500k. Your 800 tok/s figure is met by prefill, which peaked at 2963 tok/s. Generation is a different quantity and is bandwidth-bound: 800 tok/s single-stream is not reachable on this hardware at any context.

Generation speed is roughly flat with window size, but falls with actual depth: 36 tok/s at 120k tokens deep, ~10 tok/s at 431k.

Retrieval quality (bin/needle)

Window size means nothing if the model cannot find anything in it.

Depth of prompt	Needle found
151k tokens, depths 25/50/90%	3/3 PASS
120k tokens, depths 25/75%	2/2 PASS
431k tokens (YaRN 2x), depth 50%	PASS
431k tokens (YaRN 2x), depth 90%	FAIL

So: ~500k is real and usable, with a caveat. Retrieval is solid up to at least mid-depth at 431k, but degrades at extreme depth under rope scaling. The 1M max preset loads and runs, but treat it as a ceiling, not a place to hide important facts. Below 262k (fast) there is no scaling at all and quality is best.

Night profile, 30B MoE

Config	Result
128k, experts in system RAM	17.1 tok/s
262k, experts in system RAM	11.8 tok/s
experts on the 2060	crashes

The 2060 is Turing (sm_75) and lacks working kernels for this model's IQ3 expert tensors, so ggml aborts at load. EXPERTS=cpu is the default for that reason. The 2060 still contributes to the day profile, where it holds ordinary weight layers.

The 30B is dense-KV (51 KiB/token vs the 9B's 17 KiB/token), so it does not reach 500k. Use the 9B for huge windows and the 30B for hard code.

Why the 9B can do 500k at all

bin/kv-calc reads the GGUF and computes this:

architecture      qwen35
attention         hybrid (1 full-attention layer every 4; 8/32 pay KV,
                  the rest keep a fixed-size recurrent state)
KV per token      17.00 KiB

Only 8 of 32 layers pay per-token KV cost. That is what makes a 500k window affordable here, and why the same trick fails on the dense 30B.

The role each piece plays

4070 Super  workhorse: all compute, attention, and the KV cache
2060        holding card: ~3/12 of the weight layers (-ts 9,3),
            freeing 4070 VRAM for a bigger window
system RAM  CUDA unified-memory spill ("PVRAM") + prefix cache
NVMe        model weights (mmap) + hot KV snapshots
HDD         cold KV archive

The 2060 cannot be "VRAM only": whichever card holds a tensor runs that tensor's math. Giving it weight layers while the 4070 keeps the latency-critical KV path is the closest achievable version of that idea, and it is what the -ts 9,3 split does.

PVRAM

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 is the real mechanism. It uses cudaMallocManaged, so an allocation larger than free VRAM pages into system RAM via the GPU's hardware pager instead of failing. Measured effect: 1M context went from cudaMalloc failed: out of memory to loading and running at 62 tok/s.

A user-space "file on the SSD as VRAM" would be strictly worse: every access would trap to a filesystem read. The SSD's correct job is snapshots, not live tensors.

KV snapshots: pay the prefill once

bin/kvcache adds a tiered cache over llama-server's slot API.

bin/kvcache save    repo-name        # after a big ingest
bin/kvcache restore repo-name        # seconds, not minutes
bin/kvcache list                     # tiers, sizes, last use
bin/kvcache evict                    # enforce the NVMe budget
bin/kvcache promote repo-name        # HDD -> NVMe

Measured on a 93k-token prefill:

Operation	Time
Cold prefill	35 s
Snapshot save (1.56 GB)	3.2 s
Snapshot restore	0.5 s
Demote to HDD / promote back	0.3 s

Tiering: hot on NVMe with a 60 GB LRU budget (KV_HOT_BUDGET_GB), cold on the external HDD (KV_COLD_DIR, default /run/media/sponge/HDD1/kv-archive).

Honest limitation of disk restore

Restore genuinely reloads the tokens (n_restored: 93049 in 0.5s), but llama-server does not then treat them as a reusable prefix: the next matching request still re-prefills, reporting cached_tokens: 0. Restore repopulates slot state, not the prompt-cache index.

What does work, and is what actually matters for agent loops:

Scenario	Result
Repeat request, same server	35 s -> 0.3 s, cached=93043
Append a turn to the conversation	0.3 s, cached=93049
Restore from disk, then request	re-prefills, cached=0

So a long-running agent conversation is very fast, because in-memory prefix caching does the heavy lifting. Cross-restart prefill reuse is the one piece that llama.cpp does not currently deliver. --cache-reuse is also auto-disabled by this hybrid model ("cache_reuse is not supported by this context").

Slot context is capped by GGUF metadata

llama-server silently clamps every slot to the model's declared n_ctx_train (see n_ctx_slot() in tools/server/server-context.cpp). Passing -c 655360 plus YaRN was not enough: slots stayed at 262144 and big prompts were rejected with "exceeds the available context size".

bin/gguf-set-ctx patches that one metadata field in place (instant, no re-download, reversible). bin/llm-day calls it automatically to match the selected preset.

Gaming

bin/llm off          # cleanest: both GPUs free
GAME=1 bin/llm-day   # 9B moves to the 2060, 4070 untouched

GAME=1 runs a 131k window on the 2060 alone. Note the 2060 cannot hold the larger Q4_K_M quant plus compute buffers, which is why the day model is IQ3_M.

Optimizing a GGUF

bin/ggsquash rewrites a model into a faster, smaller, drop-in GGUF by fusing Q/K/V into one matmul and shrinking oversized tensors. Measured +3.1% generation with correctness held. See GGSQUASH.md for the full method, numbers, and limits.

Layout

bin/      llm, install, llm-day, llm-night, llm-status, cook, kvcache, kv,
        kv-calc, ctx-probe, needle, bench-stack, bench-experts, gguf-set-ctx
models/ GGUF weights
kv-slots/ hot KV snapshots + index.json
jobs/   queue/ and done/ for overnight work
logs/     build, server, probe, and benchmark logs
systemd/  unit files (bin/install copies these into ~/.config/systemd/user)
src/    llama.cpp source + CUDA build (sm_75 + sm_89)

Overnight cooking

cp jobs/examples/sample-jobs.jsonl jobs/queue/
bin/llm cook

bin/cook takes .jsonl (one chat body per line) or .md/.txt (whole file as one prompt), writes results to jobs/done/, and snapshots the slot after each job so a 4am crash does not replay a long prefill.

Tools worth knowing

bin/kv-calc models/<model>.gguf      # KV cost per token, feasibility per window
bin/ctx-probe day                    # find the real context ceiling
bin/needle --tokens 430000           # retrieval quality at depth
bin/bench-stack                      # reproduce every number above

Practical recommendation

• Default to WINDOW=long (524k). It hits 40+ tok/s and retrieval holds.
• Drop to WINDOW=fast (262k) when you want best quality and no rope scaling.
• Use WINDOW=max (1M) only as a ceiling for bulk ingest.
• For repo-scale work, prefer agent file-reading tools over stuffing 500k tokens. A long window is a fallback, not a substitute for retrieval.