llama.cpp and GGUF

Run a model on this machine: no account, no per-token cost, nothing leaving the disk. llama.cpp does the running; GGUF is the single-file format it reads; the quantisation level is the dial that trades quality for memory.

Two jobs live here. Finding the right file on the Hub, which is mostly a research task, and running it, which is mostly an arithmetic one about how much memory the user has.

Finding the file

The Hub's web pages are the map, but the tree API is the territory. A model card can list quants that were renamed, split or never uploaded. Confirm against the API before handing anyone a command.

1. Find candidate repositories, with web_fetch:

   https://huggingface.co/models?apps=llama.cpp&sort=trending
   https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
   https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
   

   apps=llama.cpp is the filter that matters: it excludes everything with no GGUF.

2. Open the repository's local-app view, which carries the maintainer's own command:

   https://huggingface.co/<repo>?local-app=llama.cpp
   

   If the snippet is readable, take the command and the recommended quant verbatim. The Hardware compatibility block is better than any general table here, because it was written against these specific files.

3. Confirm against the tree API, which lists what actually exists:

   https://huggingface.co/api/models/<repo>/tree/main?recursive=true
   

   It returns a JSON array; keep entries whose type is file and whose path ends in .gguf. path and size are the truth for filenames and bytes.

   Three kinds of file show up together and must not be confused:

   • the quantised checkpoints, which is what you want;
   • mmproj-*.gguf, the vision projector for a multimodal model, which is loaded ALONGSIDE the main file and is useless alone;
   • shards such as BF16/ or *-00001-of-0000N.gguf, an unquantised model split across files.

4. Build the command. Shorthand when the quant tag is standard, exact file when the repository names things its own way:

   llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M
   llama-server --hf-repo <repo> --hf-file <exact-name-from-the-tree-api.gguf> -c 4096
   

5. Suggest converting from Transformers weights only when the repository publishes no GGUF at all. It is a long job and someone has usually already done it: search for <model name> GGUF first.

Report the label exactly as the repository writes it. UD-Q4_K_M and IQ4_NL_XL are real, specific names. Normalising one to Q4_K_M produces a command that downloads nothing.

Install

brew install llama.cpp      # macOS, Linux
winget install llama.cpp    # Windows

From source, when you need a specific backend compiled in:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

Verify with llama-cli --version before anything else. A missing binary and a failed model load produce different errors, and confusing them costs an hour.

The arithmetic that decides everything

Before recommending a quant, work out what fits. Roughly:

• The file size is the floor, not the total. Add context on top.
• Context costs memory too, and it grows with -c. A large context window on a small machine is what turns a working setup into an out-of-memory crash halfway through a conversation.
• Leave headroom. A model whose file is 90% of available RAM will swap, and a swapping model is slower than the CPU-only path it was supposed to beat.

Then pick:

Situation	Quant
General chat, the default worth starting from	Q4_K_M
Code or anything technical, if memory allows	Q5_K_M, Q6_K
It must fit, quality second	Q3_K_M, or an IQ variant
Memory is not a constraint	Q8_0

IQ quants are smaller at equal quality but need more compute to unpack, so on a slow CPU they can be the wrong trade. Below Q3 the model degrades in ways that look like stupidity rather than compression.

If the local-app view named a quant for the user's hardware, prefer it over this table.

Serving

llama-server -hf <repo>:<QUANT> -c 4096 --port 8080

It exposes an OpenAI-compatible API, which is what lets existing clients talk to it unchanged:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"say hello"}]}'

A JSON reply with a choices array means the model is loaded and serving. Anything else, read stderr: llama-server reports the load there, including the layer count it offloaded and the memory it took.

Python bindings

pip install llama-cpp-python

CMAKE_ARGS="-DGGML_CUDA=on"  pip install llama-cpp-python --force-reinstall --no-cache-dir
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

The plain install is CPU-only. GPU support is compiled in, so switching backends means reinstalling with --force-reinstall --no-cache-dir: without those flags pip serves the cached CPU wheel and the GPU is silently never used.

from llama_cpp import Llama

llm = Llama(model_path="C:/models/model-q4_k_m.gguf",
            n_ctx=4096, n_gpu_layers=35, n_threads=8)

reply = llm.create_chat_completion(
    messages=[{"role": "user", "content": "what is a GGUF file"}],
    max_tokens=256,
)
print(reply["choices"][0]["message"]["content"])

for chunk in llm("explain quantisation:", max_tokens=256, stream=True):
    print(chunk["choices"][0]["text"], end="", flush=True)

hub = Llama.from_pretrained(repo_id="bartowski/Llama-3.2-3B-Instruct-GGUF",
                            filename="*Q4_K_M.gguf", n_gpu_layers=35)

n_gpu_layers is how many transformer layers move to the GPU. 0 is CPU-only, -1 is all of them, and a number too high for the available VRAM fails at load rather than falling back.

Reporting a discovery

Repo: <repo>
Recommended: <label exactly as published> (<size>)
Command: llama-server -hf <repo>:<label> -c 4096
Also available:
  <filename>  <size>
  mmproj-<...>.gguf  <size>   (vision projector, load with the main model)
Sources:
  https://huggingface.co/<repo>?local-app=llama.cpp
  https://huggingface.co/api/models/<repo>/tree/main?recursive=true

Give the sizes. "Q4_K_M is recommended" is not actionable; "Q4_K_M, 2.0 GB" is.

Traps

• A quant on the model card that does not exist as a file. Cards go stale. The tree API does not.
• Downloading before checking the size. These are gigabytes on someone's connection and disk. State the number first.
• mmproj-*.gguf treated as the model. It is the vision half and produces nothing alone.
• Normalising a repository's own quant label. It is part of the filename.
• Reinstalling llama-cpp-python without --no-cache-dir. pip serves the cached CPU build and the GPU flags are silently ignored.
• Raising -c to the model's maximum by reflex. Context is memory. A 128k window on a laptop is an out-of-memory error waiting for a long conversation.

Failure modes

llama-server: command not found. Not installed, or not on PATH after a source build: the binaries land in build/bin/.

The model loads, then the process dies. Out of memory. Lower -c first, since it is free to change, then drop to a smaller quant.

Generation is far slower than expected, and the GPU is idle. n_gpu_layers is 0, or the wheel is the CPU build. Read llama-server's stderr: it prints how many layers were offloaded.

ValueError: Model file not found. The path or the filename glob matches nothing. Check it against the tree API output rather than guessing at the pattern.

The download stops partway, repeatedly. Large files over an unstable connection. Fetch it with curl -C - to resume, and point --hf-file at the local path instead.

Output is fluent but wrong in a way a smaller model would not be. The quant is too aggressive for the task. Move up one level before concluding the model is unsuitable.