GEARBX
Documentation

Python Library

Gearbx is an entropy-routed dynamic quantization engine for LLM inference. It reads the model's output entropy at each generation step and routes the next forward pass through the right precision tier: 4-bit for filler, 8-bit for moderate reasoning, fp16 for the hardest tokens. No retraining, no fine-tuning.

Installation

Pick the extras that match your hardware. The base install runs on CPU and Apple Silicon (MPS).

shell
# Base (CPU / Apple Silicon MPS)
pip install gearbx

# Apple Silicon with MLX backend
pip install "gearbx[mlx]"

# NVIDIA CUDA (bitsandbytes + Triton fused kernels)
pip install "gearbx[cuda]"
BackendBase dtypeQuantized path
CUDA4-bit NF4 (bitsandbytes)Fused Triton kernels on packed int4/int8
MPSfp16Dequant cache after first forward
MLXfp16Native MLX quantization
CPUfp32Packed int storage (testing only)

Quickstart

Load a model, generate, inspect the gear distribution.

python
from gearbx import GearbxModel

gbm = GearbxModel("meta-llama/Meta-Llama-3-8B-Instruct")

result = gbm.generate("Prove that sqrt(2) is irrational.")

print(result.text)
print(result.gear_stats)    # {'low': 0.12, 'mid': 0.61, 'high': 0.27}
print(result.tokens_per_sec)

Thresholds auto-calibrate from the first prompt's prefill pass. No manual tuning needed to get started.

GearbxModel

Full per-token gear shifting with real weight swaps. Uses HuggingFace transformers and reaches into attention layers to hot-swap quantized weights mid-generation.

python
GearbxModel(
    model_name: str,
    num_attn_layers_to_manage: int = 16,
    monitor_window: int = 5,
    high_thresh: float = 3.5,
    low_thresh: float = 1.8,
    device: str = "auto",          # auto | mps | cuda | cpu
    prompt_classifier=None,
)
ParamDefaultDescription
model_namerequiredAny HuggingFace causal LM identifier.
num_attn_layers_to_manage16How many attention layers get dual-precision treatment. More layers, more memory savings, more overhead.
monitor_window5Rolling-window size for entropy smoothing.
high_thresh3.5Entropy (bits) above which the model upshifts to fp16. Auto-scaled by vocab size.
low_thresh1.8Entropy below which the model downshifts to 4-bit.
device"auto"Detection order: MPS > CUDA > CPU.
prompt_classifierNoneOptional cold-start gear estimator. See PromptClassifier.

generate()

Manual autoregressive loop with KV caching. At each step: forward pass, read entropy, route the next step's precision.

python
result = gbm.generate(
    prompt: str,
    max_new_tokens: int = 512,
    temperature: float = 1.0,
    top_p: float = 0.9,
    min_p: float = 0.0,            # > 0 uses min-p filtering (6x faster on MPS)
    do_sample: bool = True,
    use_cache: bool = True,        # False for strict mixed-precision validation
    min_gear_duration: int = 1,    # raise to 4-8 to reduce gear oscillation
)
ParamDefaultDescription
max_new_tokens512Generation length cap.
min_p0.0Min-probability filter relative to top token. When > 0, skips the 128k-element sort that top-p needs. Typical: 0.05 to 0.1.
use_cacheTrueKV caching. Set False to avoid mixed-precision KV entries from gear shifts.
min_gear_duration1Minimum tokens before allowing a gear change. Higher values reduce KV cache noise on long sequences.

GenerationResult

Every generate() call returns this dataclass with full introspection data.

python
@dataclass
class GenerationResult:
    text: str                      # decoded output
    gear_trace: list[str]          # gear per token: ['mid','low','low','high',...]
    entropy_trace: list[float]     # entropy (bits) per token
    gear_stats: dict[str, float]   # fraction of tokens per gear
    total_tokens: int
    elapsed_sec: float
    tokens_per_sec: float

OllamaGearbx

Black-box adapter for any OpenAI-compatible local server (Ollama, llama.cpp, vLLM, LM Studio). Entropy telemetry is real, reconstructed from top_logprobs. Gear routing is per-prompt rather than per-token, since the server's weights cannot be swapped over HTTP.

python
from gearbx import OllamaGearbx

gbm = OllamaGearbx(
    gear_models={
        "low":  "llama3.2:1b",
        "mid":  "llama3.2:3b",
        "high": "llama3.1:8b",
    },
    low_thresh=0.4,        # narrower range for top-k truncated entropy
    high_thresh=1.2,
)

r = gbm.generate("Prove sqrt(2) is irrational.", max_new_tokens=200)
print(r.text)
print(r.gear_stats)
print(r.dispatched_model)   # which model actually served the prompt

# Streaming with per-token telemetry
for chunk in gbm.generate_stream("Explain entropy."):
    print(chunk.token, chunk.entropy, chunk.gear)

Use lower thresholds for the Ollama backend (low=0.4, high=1.2) because top-k truncated entropy has a narrower range than full-vocab entropy.

EntropyMonitor

The core routing primitive. Computes Shannon entropy (bits) from logits, smooths over a rolling window, maps to a gear. Usable standalone.

python
from gearbx import EntropyMonitor

monitor = EntropyMonitor(
    window=5,
    low_thresh=1.8,
    high_thresh=3.5,
    vocab_size=128256,     # auto-scales thresholds across model families
    hysteresis=0.1,        # margin before gear flips, prevents oscillation
    warmup_tokens=16,      # auto-calibrate from observed distribution after N tokens
)

gear = monitor.update(logits)   # returns 'low' | 'mid' | 'high'

When vocab_size is set, thresholds scale by log2(vocab_size) / log2(32768) so the same values transfer across Llama, Qwen, Gemma, Mistral, and others. GearbxModel passes this automatically.

StatisticalPromptClassifier

Optional cold-start heuristic. Scores prompt complexity from math symbols, code keywords, length, and average word length to pick an initial gear before the entropy window fills. Runs in O(n) with no model inference.

python
from gearbx import GearbxModel, StatisticalPromptClassifier

clf = StatisticalPromptClassifier()
gear = clf.classify("Integrate x^2 from 0 to 1.")   # 'high'

gbm = GearbxModel("meta-llama/Meta-Llama-3-8B-Instruct",
                  prompt_classifier=clf)

Utilities

Helpers exported from the top-level package.

python
from gearbx import (
    detect_device,        # 'mps' | 'cuda' | 'cpu'
    memory_usage,         # process RSS
    vram_usage,           # GPU memory in use
    gear_report,          # human-readable gear distribution
    format_gear_trace,    # colorized per-token gear string
    memory_comparison,    # bytes/param across precision tiers
    HAS_TRITON,           # fused CUDA kernels available?
    HAS_MPS_NATIVE,       # native Metal kernels available?
)

Precision Tiers

LOW 4-bit Filler tokens. 0.5 bytes/param, 75% memory saved vs fp16.
MID 8-bit Moderate reasoning. 1 byte/param, 50% memory saved.
HIGH fp16 Hard reasoning. Original weights restored, full fidelity.

Threshold Tuning

Thresholds map averaged entropy to a gear. They auto-scale by vocab size and auto-calibrate from the prefill distribution at the p40/p60 percentiles. Post-calibration caps prevent inflation from instruction-format tokens: high ≤ log2(vocab) * 0.18, low ≤ log2(vocab) * 0.06.

Backendlow_threshhigh_threshWhy
Transformers (full vocab)1.83.5Full Shannon entropy range.
Ollama (top-k logprobs)0.41.2Truncated entropy, narrower range.

Research Paper

The full method, calibration scheme, and evaluation are written up in the Gearbx paper.

Research Preview

Entropy-Routed Dynamic Quantization for LLM Inference

Jpdz Labs · OpenReview

Read on OpenReview →