Local models: llama.cpp, Ollama, LM Studio, vLLM

Running the Dungeon Master on your own GPU: the recommended model and its exact settings, the llama.cpp command, the Ollama Modelfile, LM Studio and vLLM, thinking mode and the reasoning budget, why presence penalty is pinned, and what to expect from a turn.

Desktop appServer

A local model keeps every word of the campaign on your machine. It needs a GPU with enough memory for the model at a real context window, and it needs the settings on this page: the difference between a Dungeon Master that rolls dice and one that only talks about them is a handful of flags.

The easy way: the desktop app

The desktop app's Story AI screen has a Local AI on this machine door that measures your hardware, downloads a model sized to it, installs a pinned llama.cpp server with every setting below already applied, and runs it for you. If you host from the desktop app, use that and skip the rest of this page. See Story AI on the desktop. On a self-hosted server, the guided setup looks for llama.cpp, Ollama, LM Studio, vLLM, KoboldCpp, TabbyAPI and Jan running on the machine and lists their models, the documented default preselected.

The recommended model

Qwen3.6 35B-A3B (a mixture-of-experts model, so it runs faster than its size suggests) at Q8 if it fits, Q4 otherwise, served by llama.cpp with a 64K context window. Smaller machines fall back to Gemma 4 12B. The critical settings, which port to any server:

Setting	Value	Why
Context	65536	A small window silently truncates the prompt (sheets, scene, story) and makes the DM loop and repeat.
Tool calling	On (`--jinja`)	Dice and every sheet change depend on it.
Thinking	Enabled for tool decisions	Requested per call by the app; a non-thinking model calls tools about one turn in five.
Reasoning budget	1024 to 2048 tokens	Uncapped, a hard turn can spiral for minutes.
Temperature, top-p, top-k, min-p	0.7, 0.95, 20, 0	The model's recommended sampling.
Presence penalty	0	A penalty suppresses tool calls; the app pins it and no profile exposes it.

llama.cpp

llama-server -m Qwen3.6-35B-A3B-Q8_0.gguf \
  -c 65536 --jinja \
  --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
  --temp 0.7 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --reasoning-budget 2048 \
  --port 8001 --alias qwen3.6-35b

This is the server's default backend: provider Custom, URL http://127.0.0.1:8001/v1, model qwen3.6-35b, so nothing needs to be set. If you start llama-server with --api-key, put the key in OPENAI_COMPAT_API_KEY. The server reads the real context window from llama.cpp, so fixing it there is enough.

Ollama

The repository ships a Modelfile that bakes the same settings in:

ollama pull qwen3.6:35b-a3b-q8_0
ollama create qwen3.6-dm -f models/qwen3.6-dm.Modelfile
# backend URL http://127.0.0.1:11434/v1, model qwen3.6-dm

Point the Custom backend at Ollama's OpenAI-compatible endpoint. The older Local (Ollama) provider talks to Ollama natively and offers a curated Gemma 4 list (from gemma4:e2b-it-qat at 4.3 GB to gemma4:31b-it-qat at 19 GB); the 26B mixture-of-experts build is the sweet spot with 24 GB or more of memory.

LM Studio and vLLM

Both expose an OpenAI-compatible server. Load the model, turn tool calling on (LM Studio: the model must be one with a tool-capable template; vLLM: --enable-auto-tool-choice with the right parser), set the context to 65536, and use the server's URL as the Custom backend. The server treats them as local and sends every sampling parameter; the context is assumed to be 16K unless the endpoint reports it, so set OPENAI_COMPAT_CONTEXT=65536.

From Docker

The container reaches the host as host.docker.internal: http://host.docker.internal:8001/v1 for llama.cpp, :11434 for Ollama. Or use host networking on Linux so the plain 127.0.0.1 defaults work. See Run with Docker.

What a turn costs

A turn is two or three calls: a tool-decision call with thinking (about 50 to 100 seconds on one consumer GPU), the tool results, and a streaming narration call without thinking so the words appear smoothly. Expect 1.5 to 3 minutes for a full turn, combat at the longer end. The descriptions of the Dungeon Master's tools take about 27,000 tokens of every call; the prompt budget counts them (the Context tab shows a Tool definitions row), which is why 64K is the sensible floor. A request that lands on a connection the model server just closed is asked again once. Setting DM_THINKING=0 forces thinking off everywhere: fast turns, unreliable tools, a fallback rather than a normal mode.

Checking it

Admin, Server settings, Text model, Test backend, or from a shell:

node scripts/probe-openai-backend.mjs --url http://127.0.0.1:8001/v1 --model qwen3.6-35b

It streams a line, makes a real function call with exact arguments, and continues after the tool's output. A backend that passes rolls dice at the table.

ComfyUI, Kokoro and the text model all want the same memory. On an integrated GPU the server queues media jobs behind each other; a scene picture arrives after its paragraph, and a turn may wait behind a render. Two GPUs, or pictures from an OpenAI key, keep the story moving.