Can a 125B open model do my small dev work on one GPU?
I have a home server with 128 GB of RAM and no usable GPU. Before buying a card, I wanted to know two things:
- Is a local open-weight model good enough for the small dev and review work I hand to Claude Code today?
- If so, which GPU (or CPU upgrade) gives the best speed for that box?
So instead of buying first, I rented GPU boxes by the hour and measured. This post is the data so far, with a short narrative around it.
The setup
Model. Qwen3.8-Flash-Next, released by Alibaba in August 2026 as an open-weight preview of the Qwen4 architecture. It is a mixture-of-experts model: 125B parameters in total, about 6B active per token. That shape is what makes it interesting for a box with lots of RAM and a modest GPU: most of the weights sit idle on any given token.
Quantizations. Two GGUF builds:
| Quant | Size on disk |
|---|---|
| IQ3_S (≈3-bit) | 84 GB |
| UD-Q4_K_XL (≈4-bit) | 111 GB |
Engine. Strata, which is built for exactly this case. It keeps the attention layers and a cache of the most used experts in VRAM, pins the full expert set in system RAM, and serves every expert one of three ways on each token:
- from the VRAM cache (fast),
- read by the GPU from pinned RAM over PCIe,
- computed by a pool of CPU cores.
The last two both spend system memory bandwidth, which turns out to be the whole story for speed.
All runs used 128K context, int8 KV cache, one request at a time, and MTP speculative drafting.
Harness. Claude Code, pointed at the local server, replaying real work from one of my side projects, a real-estate listings app. Every task is something that was already done and merged, so I know what a good answer looks like. Each run had a time cap (90 min, raised to 180 for the slow 4-bit runs). The scoring is explained with the results below.
Hardware. Rented on Vast.ai:
- RTX 5070 Ti (16 GB) + EPYC 7282 (8-channel DDR4)
- RTX 5070 Ti (16 GB) + Ryzen 9 5950X (2-channel DDR4)
- RTX 3090 (24 GB) + EPYC 7742 (8-channel DDR4)
My server is a Ryzen 7 3800X with 2-channel DDR4-3200, so the 5950X box is the closest match.
Quality: is it good enough?
What the model was asked to do
There are two kinds of task: writing a change (dev) and reviewing someone else’s change (review).
Dev tasks. Each one starts from a pull request that was already merged. The model gets the code as it was before that PR plus the ticket that led to it, and has to make the change itself. It never sees the real PR.
- Task A: which price a listing shows. Some listings have no sale price but do have a rent price. The ticket asked which price the listing card should show in that case. The real fix also made the price filters and the sort order follow the same rule, so a listing appears where its card says it should.
- Task B: zero isn’t “unknown”. Some sources record a home’s room counts as 0 when they don’t know them. The ticket asked for those zeros to be shown as “unknown” for homes, while keeping real zeros for things like commercial lots where 0 is a plausible answer.
How a dev task is scored. The real PR came with its own tests. Those tests are held back while the model works; that’s what I mean by hidden tests. They weren’t written for this experiment: they’re the tests that shipped with the change, and the model had no way to read them. When the model is done, the scorer adds those tests to the model’s version of the code and runs the whole test suite. Two numbers come out of it:
- Hidden tests passed, e.g. “3/5”: of the hidden tests that fail on the original code (so they test the new behaviour), how many pass on the model’s version.
- Regressions: tests that already passed before the change and fail after it. That’s the model breaking something that used to work.
A hidden test can also fail for a reason that isn’t a bug: if the model gave a function a different name than the real PR did, the test can’t find it. Where a miss was like that, I say so below.
Review task. Here the model plays reviewer. It gets a pull request exactly as it looked when it went up for review, and has to say whether it’s ready to merge or needs fixing. The PR added a retry-with-backoff mechanism: after a failure, wait longer and longer before trying again. It had one real bug, which the original review caught at the time: work that had queued up before a failure could skip the backoff check, so with two or more cities being fetched at once, the waits weren’t always respected. It’s a subtle concurrency bug, the kind that only shows up when several things are happening at the same time. The score is simply whether the reviewer found that bug. “1/3” means it was run three times and found it once.
Results
| Model | Dev task A | Dev task B | Review: found the bug |
|---|---|---|---|
| Flash IQ3_S, local | 3/5, 0 regressions, 64 min | 8/9, 0 regressions, 87 min | 0/1 |
| Flash Q4, local | 3/5, 0 regressions (same two misses), hit the 90 min cap | 8/9, 0 regressions, hit the 180 min cap | 0/2 |
| Claude Haiku (low / high effort) | 5/5, 23–27 min | 8/9, 3 regressions | 1/3 / 2/3 |
| Claude Sonnet (low / high effort) | 5/5, 12–41 min | 8/9, 13–19 min | 2/3 / 3/3 |
What I read from this:
- On a focused change, local Flash is Sonnet-level. Task B: same hidden-test score as Sonnet, and cleaner than Haiku, which broke three existing tests.
- Task A’s two misses are a judgement call the model made, not a quant artifact. Flash got the card right but left the filters and sorting on the old rule, so a card showing a rent price could still sort as if it had no price. It even said so openly in its report; it was just the wrong call. Both quants made the identical choice and failed the same two tests.
- Task B’s one miss is the naming kind. Every model scored 8/9. For Flash, the failing test expects a name that only the real PR defines: the code works, it just has a second definition of something the original keeps in one place.
- Review is its weak spot. It was thorough (live probes, mutation tests, repeated stability runs) but never found the subtle bug in three local tries. Haiku and Sonnet found it in some runs.
- 4-bit bought nothing measurable over 3-bit. Same scores, same failures. Early on I thought 4-bit fixed task A, but that 5/5 came from Alibaba’s hosted “Flash” model, which is not these weights.
A caveat I want to be honest about: this is one or two runs per cell. It rules out a large quality gap between the quants; it does not prove they are equal. A KL-divergence comparison or a few repeats per task would tighten it.
Speed: tokens per second
Decode speed from Strata’s own logs, one stream. “From VRAM” is the share of routed experts served from the GPU cache (cache hits divided by all routed experts; Strata’s printed hit rate leaves out the PCIe reads and flatters a small card).
| Box | Quant | Config | Decode tok/s | From VRAM |
|---|---|---|---|---|
| 5070 Ti + EPYC 7282 (8-ch) | IQ3_S | stock | 77 (p10 69, p90 83) | ~75% |
| 5070 Ti + 5950X (2-ch) | Q4 | stock, 15 CPU workers | ~31 | 48–63% |
| 5070 Ti + 5950X (2-ch) | Q4 | 8 cores | ~32 | similar |
| 5070 Ti + 5950X (2-ch) | Q4 | 8 cores, RAM capped at 52 GiB | 22–24 | — |
| 3090 + EPYC 7742 (8-ch) | Q4 | stock, 32 CPU workers | 74–77 | 66–77% |
| 3090 + EPYC 7742 (8-ch) | Q4 | 8 cores | 55–60 | ~70% |
| 3090 + EPYC 7742 (8-ch) | Q4 | 8 cores, RAM capped at 52 GiB | 36–39 | — |
Prefill on a 26K-token prompt: about 1,600 tok/s for IQ3_S on the 5070 Ti/EPYC box, and roughly 2,500 tok/s for Q4 on the 5070 Ti/5950X box (estimated from wall time).
GPU power while working: the 5070 Ti ran IQ3_S at 155 W median (247 W peak); with Q4 on the 5950X box it sat at 109 W median, because it spent most of its time waiting on the CPU side.
What decides the speed
- The host’s memory system matters more than the card. The same 4-bit model ran at 31 tok/s with a 5070 Ti on a 2-channel desktop and at 76 tok/s with a 3090 on an 8-channel EPYC. Every expert that misses the VRAM cache is paid for in system memory bandwidth.
- On 2-channel memory, cores stop mattering early. 16 cores and 8 cores gave the same speed on the 5950X box: memory bandwidth was the wall. On the 8-channel EPYC, cutting from 32 workers to 8 cores cost a quarter of the speed.
- More VRAM helps, with diminishing returns. Going from 16 GB to 24 GB roughly doubled the expert cache (2,623 → 4,901 of 24,576 slots) and moved VRAM coverage from ~57% to ~71%. Expert routing is long-tailed, so each extra gigabyte is worth less than the one before.
- Don’t starve the RAM. Capping pinned RAM below the full expert set cost 25–35%: the unpinned experts lose the GPU’s PCIe path and all land on the CPU. The 4-bit model needs about 88 GB of RAM for Strata to pin everything.
- Smaller quants move fewer bytes. IQ3_S experts are ~25% smaller, so more fit in VRAM and each miss costs less. That is consistent with IQ3_S on a 16 GB card matching Q4 on a 24 GB card on the 8-channel hosts, but I have not measured both quants on the same box yet.
What this means for my server
My server: Ryzen 7 3800X (Zen 2, one CCD), 2-channel DDR4-3200, PCIe 4.0 x16. I measured its memory read bandwidth at 36.9 GB/s, about 72% of the theoretical dual-channel figure. That cap comes from the CPU’s Infinity Fabric, not from the DIMMs.
None of the rows below are measured on my machine. They are estimates scaled from the rented boxes, and the 5950X box (two CCDs, Zen 3) has more usable bandwidth than my 3800X, so its 31–32 tok/s is a ceiling for my box as it is.
| Option | Quant | Decode tok/s (estimate) |
|---|---|---|
| 5070 Ti 16 GB, CPU as is | Q4 | ~25–30 |
| 5070 Ti 16 GB, CPU as is | IQ3_S | ~30–40 |
| 3090 24 GB, CPU as is | Q4 | ~35–42 |
| Either card + Ryzen 9 5950X class CPU | either | ~31–45 |
Other things on the scale: a used 3090 has no warranty and a higher power limit, but its 24 GB would also fit a dense ~27B model entirely in VRAM, which a 16 GB card cannot. If Qwen4 ships a 27B open-weight model, as Alibaba has hinted, that matters.
Where this leaves me
- Quality: good enough for focused dev tasks, not yet for review work where catching the subtle bug is the point.
- Quant: 3-bit is the better fit for a 128 GB box: same measured quality, 84 GB instead of 111 GB, and faster.
- Speed: on my 2-channel box I expect 30–40 tok/s, about half of what the rented 8-channel servers gave. Usable, not fast.
- Purchase: on hold until the numbers below are measured.
What’s next
- A matched grid on 2-channel hosts. Both quants on a 5070 Ti box and on a 3090 box, both with desktop Ryzen CPUs, with each box’s memory bandwidth measured first so the result can be scaled to my server. That turns the estimate table into measurements.
- Tighten the quality comparison between the two quants with KL divergence or repeated runs.
- Watch Qwen4. Qwen4-Max-Preview went live on Alibaba’s cloud on 11 November 2026 as a hosted-only model. Whether a Qwen4 27B or Flash ships with open weights decides whether 24 GB of VRAM is worth paying for.
Lessons from renting GPUs
- Check a box’s real download speed before installing anything. One listing advertised 749 Mbps and pulled 3 MB/s from Hugging Face, which would have been about ten hours for the model. A 30-second
curl -w '%{speed_download}'against a model shard catches it. - Check the per-GB download price, not only the hourly rate. On one box, bandwidth cost more than the GPU time.
- A 4-bit model on a 16 GB card on a 2-channel host is too slow for long agentic tasks: both dev runs used their full time cap.