Minimally-capable Qwen

2026-08-30

My day job involves too much LLM usage, and I’m quite happy not bringing that home. In particular, I absolutely do not want to pay any LLM provider any of my own money.

With that in mind, if there’s a reasonably-scoped change that’s suitable for an LLM which would be convenient for an open-weight model to do on my behalf, on my own hardware, can one actually get it done? I’m thinking of tasks that don’t demand architecture, or thinking; more paint-by-numbers while I’m doing the dishes or in the garden.

So far none of my previous attempts have succeeded — the models talk themselves into corners, they seem to approach the problem but fail make edits, they crash in the loop, whatever — but with some poking and prodding, I’ve finally had minimal viable success.

My personal machine is an M3 Max with 64GB of unified memory. This is enough to hold <30GB of model, its KV cache, and all the dev environment without my laptop keeling over. Qwen3.8 27B was released a few weeks ago and seems like it oughta do the job.

It was really struggling to make edits at first, getting horribly confused by indentation, but reading suggested a too-small quant could be the cause (I guess different runs of spaces might end up getting collapsed into one token?). Ollama’s qwen3.8:27b-mlx turns out to use NVFP4, so I tracked down what looked like a reasonable Q8. I got really hung up on using an MLX-format model at first but it seems like the GGUF ones should be at least just as good on Apple Silicon.

So we use hf.co/unsloth/Qwen3.8-27B-GGUF:Q8_0 as the model ID. To support Qwen3.8’s reasoning levels in e.g. Pi, we need a template hack to prevent xhigh being sent as max:

{
  "id": "hf.co/unsloth/Qwen3.8-27B-GGUF:Q8_0",
  "reasoning": true,
  "thinkingLevelMap": {
    "off": null,
    "minimal": null,
    "low": "low",
    "medium": "medium",
    "high": null,
    "xhigh": "xhigh",
    "max": null
  },
  "compat": {
    "thinkingFormat": "chat-template",
    "chatTemplateKwargs": {
      "enable_thinking": { "$var": "thinking.enabled" },
      "reasoning_effort": {
        "$if": {
          "===": [ { "$var": "thinking.effort" }, "max" ]
        },
        "then": "xhigh",
        "else": { "$var": "thinking.effort" },
        "omitWhenOff": true
      }
    }
  }
}

It takes a long time; xhigh does seem to be necessary for it to get things right, and you really don’t want to wait on it, but with clear instruction and existing structure to go by, it gets there.

Notes