Writing / 24 SEP 2026

Running a 27B model on my Mac every day

A 27-billion-parameter model, running on the Mac I already own, fast enough to back a real app. Here's the setup, the numbers, and the two problems that took longest to solve.

by xz3 min read

For the last few weeks my default language model has been a local one: Qwen3.8-27B, quantised to 4 bits, running through Apple's MLX on my Mac. It backs the local mode of Tunepick, a chat page I use from my phone, and the local side of eki. None of it needs a cloud account.

The setup

Everything lives in one folder, ~/ai/llm. One script, start.sh, starts mlx_lm's built-in server, which speaks the OpenAI chat-completions API on port 8080. Anything that can talk to OpenAI can talk to it by changing a base URL.

MODEL="mlx-community/Qwen3.8-27B-4bit"
export HF_HOME="$HOME/ai/cache"
export HF_HUB_OFFLINE=1      # weights are already local

python -m mlx_lm server --model "$MODEL" \
  --host 127.0.0.1 --port 8080 --max-tokens 8192 \
  --temp 1.0 --top-p 0.95 --top-k 20 \
  --chat-template-args '{"enable_thinking":false}'

A few choices in there are deliberate:

Thinking is a switch, not a default

Qwen models can reason out loud before answering. That's great for hard problems and terrible for anything interactive: the benchmark log below shows it spending its whole 300-token budget deliberating about whether a code block counts as "code only". So the script takes a THINK setting:

./start.sh             # thinking off — fast, interactive
THINK=low ./start.sh   # brief reasoning
THINK=1 ./start.sh     # full reasoning

Off is the default. For an app like Tunepick, where the model returns structured JSON a player is waiting for, thinking adds latency and changes nothing about the quality of a bassline.

The numbers

From my own benchmark run on this machine:

17 tokens a second is slower than a hosted model, but it's comfortably faster than reading speed, and it streams, so the first words appear almost immediately. For a Tunepick option — a few hundred tokens of JSON — that's a wait of seconds. The memory figure is the real constraint: it decides what else can run at the same time, which is a big part of why eki exists.

Problem one: phones and streaming

I wanted to use the model from my iPhone over home Wi-Fi. The model server was reachable, the page loaded, and streaming simply didn't work in iOS Safari: the answer arrived all at once at the end, or not at all.

The fix was to stop streaming across origins. A tiny Python server, serve.py, runs on port 8081 and does three things on the same origin: serves the chat page, stores conversation history in a JSON file on the Mac, and proxies every /v1/* request to the model server, forwarding the reply line by line and flushing after each one. Because each line of a server-sent-events stream is one event, reading upstream a line at a time and flushing immediately is enough to make it stream smoothly. With the page and the API on the same origin, Safari streams it without complaint.

Problem two: not exposing it to a café

A model server bound to 0.0.0.0 is reachable by everyone on the network you're on. That's fine at home and not fine in a café. So LAN access is opt-in: by default the server binds to 127.0.0.1, and only LAN=1 ./start.sh binds to all interfaces. When it does, it prints the Mac's Bonjour name and IP so I can type the address on my phone.

Was it worth it?

For my use, yes. The local model handles the everyday work — drafts, quick questions, generating Tunepick options while I'm developing — with no per-token cost and no data leaving the machine. For the hardest problems I still use a hosted model. Switching between them is a config change, because everything speaks the same API. That's the real lesson: pick one interface and make every model, local or not, sit behind it.