benchtest results
Apple M5 Max · 128 GB · 4 frameworks × 6 agent harnesses × 3 build tasks · one model resident at a time ·
the benchmark tool ·
raw data
🥇 Fastest Flash-Next
MTPLX · 84.9 tok/s
agent decode, server-measured · 105.6 peak on tetris raw
🥈 Close second
OMLX · 87.6 tok/s
Flash-Next 4-bit · 91.1 peak agent decode
⚠ The catch
MLX-VLM · 39.0 tok/s
Flash-Next gives it NOTHING — the framework decides, not the model
✓ Reliability
125/126 cells
72/72 Qwen · 53/54 Flash-Next · 1 honest thinking-cap error
Qwen 3.8 27B — median tok/s
raw (one-shot API)agent decode (server-measured)
Qwen 3.8 Flash-Next (MTP) — median tok/s
MLX-Serve is absent: its preflight needs ~112.5 GB for this build — it does not fit 128 GB. First complete Flash-Next pass on MLX-VLM.
The verdict table
| Framework | Qwen-27B raw/agent | Flash-Next raw/agent | Flash-Next gain | Cells done |
|---|
| MTPLX | 51.9 / 46.6 | 89.4 / 84.9 | ~1.7× | 36/36 |
| OMLX | 53.0 / 51.0 | 88.8 / 87.6 | ~1.7× | 35/36 |
| MLX-Serve | 52.2 / 49.6 | does not fit | — | 18/18 |
| MLX-VLM | 35.8 / 34.9 | 38.7 / 39.0 | ~none | 36/36 |
Agent rows use server-measured TGS (client tps includes tool time). QA = deterministic artifact reliability; human judgment stays with the artifacts. Full methodology, per-cell JSON and manifests in the dataset.