“A 27B model in 3.9GB” can be both accurate and misleading. The number accurately describes PrismML’s native binary language weights. It does not describe the downloadable MLX package, total multimodal assets, peak application memory, or the memory needed for a long context.
That distinction matters more than the headline. Bonsai 27B appears to move extreme low-bit inference across a real deployment boundary: a model in this parameter class can run interactively on a high-end phone. But a production decision still depends on four separate budgets: weight storage, resident memory, runtime peak memory, and task reliability.
All Bonsai performance figures below are PrismML-reported unless explicitly labeled as a calculation. As of July 28, 2026, I could not find a controlled independent reproduction using the same iPhone, model artifact, runtime, and prompt suite.
Reading time: about 8 minutes | Length: about 1,800 words
TL;DR
- The 1.125-bit arithmetic and 3.9GB native language-weight figure are internally consistent.
- The public MLX package is 5.13GB, while real peak memory ranges from 5.9GB at 4K context to about 9.4GB for the full 262K window with a compressed KV cache.
- The reported 11 tokens per second and benchmark results are vendor measurements, not controlled independent reproductions.
- The 89.5% claim is a ratio of 15-benchmark averages; task-level retention is lower for tool calling, instruction following, and vision.
Why 1.125 bits per weight is plausible
The Bonsai 27B model card describes a Binary g128 representation. Each weight stores one sign bit, while every group of 128 weights shares a 16-bit scale.
The effective rate is therefore:
1 sign bit + 16 scale bits / 128 weights = 1.125 bits per weight
Multiplying 27 billion weights by 1.125 bits gives about 3.8 decimal gigabytes before small residual parameters and packaging overhead. PrismML reports a 3.9GB native language-model footprint. The arithmetic and the claimed native weight size are internally consistent.
This is useful evidence that “1-bit” is not merely a label attached to a package with most layers stored at higher precision. PrismML says the binary representation covers embeddings, attention projections, MLP projections, and the language-model head. A small tail of normalization and scale parameters remains in higher precision.
Internal consistency is not the same as independent validation, but it answers one important question: the 3.9GB claim is not obviously incompatible with the stated representation.
3.9GB is one of several memory numbers
The model card exposes enough detail to separate the deployment layers:
| Layer | PrismML-reported figure | What it means |
|---|---|---|
| Native binary language weights | 3.9GB | The compact language-model representation |
| Published MLX package on disk | 5.13GB | Safetensors package with MLX overhead and bundled vision tower |
| MLX peak at 4K context | 5.9GB | Weights, activations, cache, and runtime overhead |
| MLX peak at 10K context | 6.3GB | Same runtime budget with a larger context |
| MLX peak at 100K context | 12.2GB | Reported without KV-cache compression |
| Full 262K window | about 9.4GB | Reported with a 4-bit KV cache |
The MLX package is larger because the current grouped low-bit format stores both a scale and a bias per group. PrismML’s native representation needs only a scale. The public MLX artifact also bundles the vision tower in the same file.
Runtime adds another layer. Activations, buffers, and the KV cache do not disappear because the weights are binary. PrismML estimates roughly 1.3GB of runtime buffers across backends before long-context growth becomes dominant.
This leads to the most important correction to the launch narrative: supporting a 262K context is not the same as fitting that full context inside the same phone budget. PrismML’s own table estimates about 9.4GB peak memory for the full window even with a 4-bit KV cache. If an iPhone app has roughly 6GB available, as PrismML’s launch post suggests for a 12GB device, then the full window and the phone deployment claim cannot be assumed to hold simultaneously.
The model can support 262K in its architecture while a specific device supports a smaller practical window. Product documentation should publish that device-specific window instead of repeating the model maximum.
The bottleneck moves from weights to the runtime
Weight compression changes the system bottleneck rather than removing it.
At short context, the binary weights dominate the budget and make phone deployment possible. As context grows, the KV cache and runtime overhead take over. During sustained generation, memory bandwidth, heat, and battery become constraints. For multimodal use, the vision tower and image-processing path add another budget.
This is why a storage screenshot is a weak deployment test. A useful test records:
- the exact model artifact and runtime commit;
- resident memory after loading;
- peak memory at the intended context length;
- time to first token and sustained decode rate;
- thermal state and battery change during a long run;
- task completion quality under the same conditions.
Once the model fits, “does it stay useful while the context grows and the phone gets warm?” becomes the real question.
What the 11 tokens-per-second result measures
PrismML reports 11.0 tokens per second for tg128 on an iPhone 17 Pro Max using its MLX Swift runtime. tg128 measures token generation over 128 generated tokens. It is not prompt-processing speed and does not describe time to first token for a long document.
The whitepaper also reports a continuous battery test:
- sustained decode: 10.82 tokens per second;
- duration: about 5.2 minutes;
- battery change: 100% to 95%;
- output: 672 tokens per 1% of battery;
- thermal state: nominal to “fair,” described as mild throttling.
This is more informative than a cold burst, but it is still a vendor test. The paper says phone energy is estimated from battery percentage rather than measured with a hardware power instrument. Five minutes also does not establish performance during a long local agent session.
The correct conclusion is narrow: PrismML has published a credible method and a measured interactive rate on one flagship phone. The result still needs independent repetition across app states, ambient temperatures, context lengths, and sustained workloads.
89.5% is a ratio of benchmark averages
PrismML evaluated the FP16 reference and 1-bit model across 15 thinking-mode benchmarks. It reports:
- FP16 average: 85.07;
- 1-bit average: 76.11;
- ratio: 89.5%.
Calling this “89.5% intelligence retained” compresses a heterogeneous result into one marketing number. The category scores show why the denominator matters:
| Category | FP16 | 1-bit Bonsai 27B |
|---|---|---|
| Math | 95.33 | 91.66 |
| Coding | 88.74 | 81.88 |
| Knowledge and reasoning | 83.15 | 73.39 |
| Instruction following | 78.47 | 65.74 |
| Agentic and tool calling | 80.00 | 66.03 |
| Vision | 72.61 | 59.57 |
Math retains about 96% of the category score and coding about 92%. Agentic and tool-calling performance retains about 83%, while vision retains about 82%. A product built around repeated structured tool calls should not use the overall 89.5% ratio as its reliability estimate.
The evaluation is useful because PrismML publishes the benchmark list, settings, per-benchmark results, and parts of the harness methodology in its whitepaper. It is still a vendor evaluation. Some runs use unseeded option shuffling, and the suite does not represent every language, domain, or multi-step application.
The missing comparison is a smaller mature model
The launch comparison asks how much of a 27B FP16 model survives extreme compression. A developer often faces a different choice:
- 1-bit 27B with custom kernels; or
- a smaller 4-bit model with a mature runtime at a similar memory budget.
Parameter count is not the outcome. The relevant metric is successful tasks per device budget.
A smaller model may load faster, leave more room for context, run cooler, and integrate with more front ends. Bonsai may preserve more knowledge or reasoning capacity. Neither advantage can be inferred from parameter count alone.
The decision should use a workload-specific evaluation. For a local agent, include multi-turn tool calls, malformed arguments, recovery after tool errors, and long-session drift. For private document analysis, measure retrieval accuracy, time to first token, and peak memory at realistic document sizes. For multilingual use, test the actual languages instead of importing an English benchmark average.
A practical acceptance test
Before shipping Bonsai 27B in an application, pin one model artifact and one runtime commit, then run four gates.
Memory gate: record package size, resident weights, and peak memory at 4K, 16K, and the product’s maximum supported context. Test text and vision separately.
Sustained-performance gate: measure time to first token, prompt processing, and decode over at least 20 minutes. Record thermal state, battery change, and variance, not only the fastest run.
Quality gate: compare Bonsai against a smaller 4-bit model within the same memory budget. Score task completion, not prose preference. Separate math, coding, instruction following, and tool use.
Failure gate: test what happens when memory pressure rises, the context is nearly full, a tool returns an error, or the app enters the background. A local model is valuable only if the surrounding application can recover safely.
Conclusion
Bonsai 27B’s strongest claim is not that 3.9GB describes the whole application. It is that an end-to-end binary representation, paired with custom kernels, makes interactive 27B-class inference possible on a phone-sized memory budget.
That is a meaningful engineering boundary. It is not yet a universal deployment verdict.
The public artifacts support the 1.125-bit arithmetic and expose unusually useful memory and benchmark details. They also reveal the trade-offs: MLX packaging is larger, long context quickly dominates memory, sustained phone performance is thermally constrained, and capability retention varies sharply by task.
Treat the release as a strong pilot candidate. Treat its performance figures as vendor measurements until the same artifact and runtime are independently reproduced.
If Bonsai is on your deployment shortlist, the next action is to run the four-gate acceptance test above against a smaller 4-bit model on the exact target device.
FAQ
Is Bonsai 27B really a 1-bit model?
Its language network uses binary weights plus one FP16 scale per 128 weights, producing an effective 1.125 bits per weight. PrismML says the low-bit representation covers the major language-model layers end to end.
Does the whole model use only 3.9GB on an iPhone?
No. The 3.9GB figure describes the native language weights. The MLX package, runtime buffers, activations, KV cache, and optional vision components require additional storage or memory.
Can the iPhone use the full 262K context?
The architecture supports 262K, but PrismML estimates about 9.4GB peak memory for the full window with a 4-bit KV cache. Practical context therefore depends on the device and its per-app memory limit.
Does 89.5% retention mean answers are 89.5% as good?
No. It is the ratio of two averages across 15 benchmarks. Individual categories retain different amounts, and real applications may have different failure modes.
What should developers compare it against?
Compare it with smaller 4-bit models that fit the same peak-memory budget. Measure sustained latency, task success, tool-call reliability, energy, and context capacity on the target device.