Building a Local AI Coding Server · Part 2 of 4
Building a Local AI Coding Server – Part 2: Two Radeon GPUs, One Stubborn Bug
The plan, and the error
My plan was to run one big coding model split across both GPUs with vLLM. It worked on one GPU and crashed on two. Final score after a lot of testing: nine theories, nine identical failures, and one bug report to AMD.
vLLM is a popular program that loads a language model onto GPUs and serves it through a web API, the same kind of API that coding tools in VS Code talk to. To use both of my 32 GB cards for one model, vLLM uses tensor parallelism: every layer of the model is split in half, each GPU computes its half, and the two cards swap partial results many times for every word generated. The library that handles that swapping on AMD cards is called RCCL (AMD's version of NVIDIA's NCCL).
I installed vLLM in a Python virtual environment using AMD's ROCm 10.1 instructions. (A virtual environment is a self-contained folder with its own Python and packages, so nothing conflicts with the system.) Then I started it on both GPUs with a tiny test model, and five seconds later:
RuntimeError: NCCL error: unhandled cuda error(AMD's libraries keep NVIDIA-style names like "NCCL" and "cuda" for compatibility, which is confusing at first. Yes, an AMD card reporting a "cuda error." You get used to it.)
A small win first: one GPU
Before the two-GPU crash, I'd already hit and fixed a simpler problem: vLLM wouldn't start even on a single GPU. The last line of the error was useless (Engine core initialization failed. See root cause above.), but searching the log for the first error found it:
fatal error: Python.h: No such file or directoryAt startup, vLLM compiles small pieces of GPU code for your exact card. Part of that is a little C program that plugs into Python, and building it needs Python's header files, which Ubuntu doesn't install by default. One command fixed it:
sudo apt install -y python3.14-dev build-essentialAfter that, vLLM served a test model on one GPU and answered a JavaScript question through its API. Lesson one: the last error line is often generic; the real cause is usually the first error in the log.
The investigation: ruling things out one at a time
I tested nine possible causes, one change at a time, and every one of them failed the same way, which ended up being the most useful result of all.
The first big step was taking vLLM out of the picture. I wrote a 10-line PyTorch script that does nothing but one all-reduce, the basic multi-GPU operation where each GPU contributes numbers and both end up with the total. It failed identically, so the problem was in RCCL, not vLLM. Each later test ran in about a minute.
I also turned on AMD's runtime logging (AMD_LOG_LEVEL=1), which finally showed the real failing step:
rocvirtual.cpp:5242: AQL dispatch failed!
HIP failure: 'the operation cannot be performed in the present state'In plain words: the GPU refused to even start RCCL's program. That's different from data getting lost on the way between the cards.
| Theory | What I changed | Result |
|---|---|---|
| Cards can't access each other's memory directly (P2P) | NCCL_P2P_DISABLE=1 | Same failure. RCCL wasn't using P2P anyway |
| CPU memory protection (IOMMU) blocks the GPU | Added iommu=pt to the kernel boot options, AMD's standard advice | Warning gone, same failure (I kept the setting) |
| GPU copy engines misbehave | HSA_ENABLE_SDMA=0 | Same failure |
| Use copy engines for the all-reduce instead | RCCL_CE_ALLREDUCE=1 | Not available on this setup |
| Too little locked memory (8 MB limit) | Raised to unlimited | Same failure |
| Shared-memory transport between the cards | NCCL_SHM_DISABLE=1 (use network sockets instead) | Same failure |
| Both transports off | SHM and P2P both disabled | Same failure |
| RDNA4 protocol bug from a related AMD issue | NCCL_PROTO=Simple, the workaround AMD confirmed there | Same failure |
| Second card is faulty | Ran a large calculation on it alone | Worked perfectly |
Each of those NAME=value settings is an environment variable, a setting a program reads when it starts. You put it in front of the command, like NCCL_P2P_DISABLE=1 torchrun ..., and it only applies to that one run, so experiments are safe.
The breakthrough: the failure follows the slot
The error was always on the second GPU, the one in the slow chipset slot from Part 1. So I asked a simpler question: does that card work at all, or only fail when the two work together?
| Test | Result |
|---|---|
| Big calculation on GPU 0 alone | Works |
| Big calculation on GPU 1 alone | Works |
| Calculation on GPU 1 with both cards visible | Works |
| Copy data GPU 0 → GPU 1, and back | Works, correct numbers |
| RCCL all-reduce between them | Fails on GPU 1 |
Then the key test: I swapped the order the GPUs are listed in (HIP_VISIBLE_DEVICES=1,0), so the chipset card became "GPU 0" to the program. If the bug were about the position number, the failure would stay put. Instead, the failure moved with the card. Every time, it was the GPU behind the chipset that refused to launch RCCL's program. The bug wasn't following a number. It was following a slot.
I ran it again two days later after several reboots, with identical results. Both cards are healthy and can exchange data. Only RCCL's own GPU program fails, and only on the card connected through the chipset. That's not something I can fix with settings. Time to hand it to the people who wrote RCCL.
Filing the bug with AMD
I reported it on AMD's GitHub (the ROCm/rocm-systems repository, where RCCL issues now go), with everything an engineer would need to reproduce it without asking me follow-up questions.
What went into the report:
- A one-paragraph summary of the symptom and the key finding: the failure follows the chipset-attached card.
- The exact environment: GPUs, slots and link speeds, motherboard and BIOS version, kernel, ROCm, PyTorch, and RCCL versions.
- The 10-line reproducer script and the exact command to run it.
- What works and what was tried, as the two tables above, so nobody suggests something I've already ruled out.
- A link to a related issue with two R9700s, noting that its workaround didn't help in my case.
- Full logs as attachments.
One step I almost skipped: cleaning the logs. They contained my computer's name, my username, and my local network address. I made redacted copies with a find-and-replace (sed) and attached only those.
Until it's fixed, vLLM can only use one of my GPUs. But I didn't build this machine to wait on a bug ticket, and the fix for my actual goal came from a different program entirely. That's Part 3.
Debugging lessons
- Read the first error, not the last. The final line is usually a generic "something failed".
- Shrink the problem. A 10-line script that reproduces the bug beats a big program that might be doing a hundred things.
- Change one thing per test, and write down each result. My table of failed theories became half of the bug report.
- Test the parts on their own. "Does each GPU work alone?" is the first question to answer, and it narrows things down fast.
- Swap things around. Reversing the GPU order turned "GPU 1 fails" into "this slot fails", which is a much more useful finding.
- My first theory was wrong, and so were the next few. That's normal; each wrong theory still narrowed things down.
- Redact before you share. Logs leak hostnames, usernames, and IP addresses.
Next: Part 3, "llama.cpp to the rescue", where one 80-billion-parameter model ends up running across both GPUs anyway.