Running Qwen 3.8 27B on an Intel Arc Pro B70

I wanted the coding agent on my MacBook to use a model running on my own hardware. The Intel Arc Pro B70 made that affordable, since it has 32 GB of VRAM for a lot less than NVIDIA charges for the same amount of memory. The downside is that almost nobody runs inference on this card, so I had to figure out every setting by testing it myself. I kept a record of all of it in a repo.

The machine is a Threadripper 3970X with 64 GB of RAM, one B70, and Ubuntu with the in-tree xe driver. One thing about this card affects pretty much every other decision: its matrix engines don't support FP8 natively.

What's running

The model is Qwen3.8-27B with GPTQ INT4 weights. Those take up about 14 GB, which leaves the rest of the card for context. On top of that there's an FP8 key-value cache, a small INT4 draft model for speculative decoding that guesses four tokens at a time, and a 131,072-token context window. It's served by vLLM on Intel's XPU stack, one user at a time.

With one user and thinking turned off, warm decode runs at about 87 tokens per second, which is faster than I can read. A cold start on a prompt close to the 124,000-token limit takes 142 seconds. Running that same prompt again from cache takes 5.

Quality first, then context, then speed

That priority order decided most of the config. For the coding work I do, a 27B model at INT4 does better than a smaller model at higher precision. Agent work fills the context up with the repo, so 128K was a must. Decode speed came last, and 87 tokens per second is good enough for me.

Context was the hard part. With a 16-bit KV cache, this card maxes out around 110,000 tokens, because the speculative decoding buffers and the model's recurrent state come out of the same memory. Storing the KV cache in FP8 almost doubles how many tokens it can hold. It costs about 15 percent on cold time to first token at 31K input, and I was happy to make that trade.

What didn't help

Doubling the prefill chunk size didn't speed anything up and cost 17,000 tokens of KV capacity. Prefill on this card is limited by the work per token, so changing how tokens get grouped doesn't help. Pinning the process to two core complexes changed throughput by a fraction of a percent. A new vLLM release passed all my checks but wasn't measurably faster, so I stayed on the pinned image. The four-token draft was worth it, though. It was 6.6 percent faster than three tokens on the same requests.

What broke

The engine crashes at startup unless /dev/dri is bind-mounted into the container. Just mapping it as a device isn't enough.

The GPU monitoring tools I expected to use don't work. intel_gpu_top is hard-coded to the old i915 driver's counters, and so is btop's Intel collector. gputop is the one that actually works with xe.

The GPU can also fault every once in a while, so I set up a watchdog to automatically catch it, save diagnostics, and restart the engine, because I don't want the inference server sitting there dead until I happen to notice.

The client side

The same repo also has the MacBook side of things: config for pi, Codex, OpenCode, and Hermes, each one tested against the server before I added it. The vLLM recipe and the baseline numbers started from SergiioB's Intel Arc Pro B70 Inference Cookbook. My repo takes that work, adapts it to this one machine, and writes down what I changed.