Building a Local AI Coding Server – Part 3: llama.cpp to the Rescue

A different way to split a model

The fix for my two-GPU problem was to stop using the part of the software that was broken. Technically not a fix. Practically, it works. llama.cpp splits a model across GPUs in a way that never touches RCCL.

A language model is a tall stack of layers. vLLM's tensor parallelism slices every layer in half, so the cards have to talk constantly. llama.cpp's layer split instead gives each card a block of whole layers: GPU 0 runs the first half of the stack, then hands a small result to GPU 1, which runs the second half. That's one small handoff per step, through ordinary memory copies, which already worked fine in my Part 2 tests.

This also explained something I'd noticed earlier: Ollama already used both of my cards without complaint. Ollama runs llama.cpp under the hood.

Building llama.cpp and testing the split

I compiled llama.cpp myself for my exact chip, which took about 15 minutes, and both GPUs showed up on the first try. Compiling means turning source code into runnable programs; doing it yourself means the program is built against the ROCm version you actually have.

sudo apt install -y cmake ninja-build libssl-dev amdrocm-core-dev10.1-gfx1201
git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
  cmake -S . -B build -G Ninja -DGGML_HIP=ON -DGPU_TARGETS=gfx1201 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 16
./build/bin/llama-server --list-devices

The last command listed ROCm0 and ROCm1, both R9700s with about 32 GB free.

Then I loaded a tiny test model with --split-mode layer and checked each card's memory directly, by PCI address. GPU 0 used 1,207 MiB and GPU 1 used 1,251 MiB: both cards were holding part of the model, which is exactly what vLLM couldn't do.

Making it always on

I turned the server into a systemd service, so it starts at boot without anyone logging in and restarts itself if it crashes. systemd is the part of Linux that manages background programs.

The setup is two small files:

• A settings file (/etc/llama-server/llama-server.env) with one line listing the model and all its options. Changing models means editing this line and restarting the service.
• A service file (/etc/systemd/system/llama-server.service) telling systemd how to run it: as my user, with GPU access, Restart=on-failure, and started at boot.

Two choices that paid off later:

• The model loads from a file on disk, not a download link, so booting never depends on the internet.
• The API always calls the model local-model (--alias local-model), whatever is actually loaded. Tools connecting to it never need updating when I swap models.

I tested it the hard way. I force-killed the server, and systemd restarted it within 15 seconds with a new process ID. Then I rebooted without logging in, connected over SSH, and the health check answered {"status":"ok"}.

The commands I use day to day:

systemctl status llama-server --no-pager    # is it running?
journalctl -u llama-server -f              # live log (Ctrl+C exits the view)
sudo systemctl restart llama-server        # after changing settings
curl http://127.0.0.1:8080/health          # quick health check

Choosing the coding model

I picked Qwen3-Coder-Next, an open coding model built specifically for agents, at 4-bit compression so it fits in 64 GB with room to spare.

A few terms first:

• Parameters ("80B") are the model's learned numbers. More usually means smarter, but also bigger and slower.
• MoE (Mixture of Experts) models only use a few small "experts" per word. Qwen3-Coder-Next has 80B parameters in total but only about 3B are active per word, so it has big-model knowledge at small-model speed.
• Quantization shrinks a model by storing its numbers with fewer bits. Q8 (8 bits) is nearly lossless; Q4 is about half the size with a small quality loss.
• Context window is how much text the model can see at once, measured in tokens (about ¾ of a word each). Coding agents need a lot, because they read whole files.

The three models I considered, as of this writing:

My instinct was to grab the 8-bit version. The math had other plans. Here are the file sizes for Qwen3-Coder-Next:

The "UD" versions from Unsloth keep the most sensitive parts of the model at higher precision, and large MoE models tend to handle compression well. A third GPU (96 GB) would make the Q8 version fit, which is on my maybe-later list.

Swapping in the real model

The 50 GB download went smoothly once I knew it was actually running. I'd piped the output into a log file, and llama.cpp hides its progress bar when it isn't writing straight to a terminal, so it looked frozen. Watching the folder grow (watch -n 10 du -sh ~/models) showed it was fine.

Then came a scare: the service failed with cudaMalloc failed: out of memory and kept restarting. The model hadn't outgrown my cards. I had left the temporary test server running in another window, still holding about 25 GB on each GPU. The culprit was me. Stopping it fixed everything, and the service loaded the model in about 10 seconds.

The final settings:

-m <model file> --alias local-model --host 127.0.0.1 --port 8080
-ngl 99 --split-mode layer -c 131072 --parallel 1 --jinja
--temp 1.0 --top-p 0.95 --top-k 40

That's all layers on the GPUs, split by layers, a 128K-token context, one request at a time, the model's built-in chat template (needed for tool calling), and Qwen's recommended sampling settings.

The tool-calling result matters most. When I gave the model a made-up count_creeps tool and asked a question that needed it, it didn't guess an answer: it asked to call the tool with {"room": "W1N1"}. That's exactly the behavior an autonomous coding agent depends on. (Yes, the test tool was already about Screeps. I have priorities.)

What I learned

• When one tool hits a wall, another may go around it. I didn't need to fix RCCL; I needed software that doesn't use it.
• Layer splitting suits a lopsided setup. My slow chipset slot barely matters when the cards only hand off once per step.
• Test small before you download big. A 600 MB test model proved the split worked before I committed to a 50 GB download.
• Do the math on memory. The model file, the context, and working buffers all have to fit, and Q8 simply didn't.
• One server at a time. Most of my scary errors were two programs fighting over the same GPU memory.
• Give the model a fixed name (local-model) so swapping models never breaks the tools that use it.

I now had a fast, private coding model running around the clock. Next: Part 4, "Giving an AI agent its own user account", where I let it write code on its own, and learn why rules alone aren't enough.