Not every token deserves sixteen bits.

Gearbx reads a language model's uncertainty at every step of generation and spends precision only where it is needed. Obvious words run on 4-bit weights. Hard ones get the full sixteen.

$npm install -g gearbx Read how it works

A model knows when it is guessing.

Every token a language model writes comes from a probability distribution over its whole vocabulary. Sometimes that distribution is a spike: after p and, the next word is q, with near certainty. Sometimes it is spread thin: after Suppose √2 = p/q, with p and q, there are several reasonable ways to go on.

The spread has a name, entropy, and it costs nothing to read: the numbers are already sitting in the model's output. Gearbx reads it after every token.

Figure 1. A proof, written one token at a time. Hover any word to see the five most likely continuations the model was choosing between. Words where it was sure (dotted, brass) need very little precision; the forks in the road (underlined, vermilion) get all of it. Distributions are illustrative.

What a bit buys.

A model's weights are usually stored as 16-bit floats. Quantization rounds each weight to one of a small set of levels so it fits in fewer bits. Eight bits gives 255 levels and barely changes anything. Four bits gives 15, and the rounding starts to show. Two bits leaves almost nothing.

Fewer bits means smaller weights to move through memory on every token, and on a laptop, moving weights is most of the work.

Bit width

Distinct levels

Mean rounding error

An 8B model weighs

Figure 2. One row of weights. Grey stems are the originals, dots are where quantization puts them, and red ticks are the rounding error. Drag the slider: the error is invisible at 8 bits and obvious at 2.

A gearbox for precision.

So there are two dials: how unsure the model is, and how many bits it gets. Gearbx connects them. It smooths entropy over a short window and maps it to one of three gears: low at 4-bit, mid at 8-bit, and high at full fp16.

Below is a real trace from Qwen2-0.5B proving a theorem. Drag the two thresholds and watch the gear choice change under you.

Prompt

Low gear

Mid gear

High gear

Average precision

Gear shifts

Figure 3. Ninety-six generated tokens, measured on an Apple M4. Bars are per-token entropy; the dark line is the smoothed value the router actually reads. Drag the dashed lines (or focus them and use the arrow keys). Every token below the brass line runs on 4-bit weights.

Four steps, every token.

Forward pass
The current gear's weights compute the next distribution. On CUDA, fused Triton kernels multiply straight from packed integers.
Read entropy
Shannon entropy of the output, in bits, averaged over a short window so one odd token doesn't jerk the gears.
Route
Thresholds pick low, mid or high. Hysteresis and a minimum dwell keep it from oscillating at the boundary.
Sample
The token is chosen and the loop turns again, possibly in a different gear.

In low gear, weights take 75% less memory than fp16.

On predictable passages, generation runs up to 1.8× faster.

Against full precision, quality on standard benchmarks drops by <1%.

Methodology, hardware and full results

Run it.

$npm install -g gearbx $curl -fsSL gearbx.jpdz.app/install | sh

Then run gearbx. Needs Python 3.10+ and a running Ollama.