The Hugging Face preview of the VocabX paper comes from Dharma-AI's production Portuguese essay grader, built on Qwen3-4B. The model was already fine-tuned for a single task, yet it still carried the original Qwen vocabulary designed for many languages and text types. That mismatch is the core of the story: a small language model paying a multilingual tax on every response.
A fixed tokenizer becomes a quiet tax
The grader receives an essay, prompt and supporting texts, then returns a JSON assessment. Every response repeats the same JSON envelope, competency names and evaluation phrases. The inherited tokenizer split those frequent spans into several tokens, and more importantly, kept 151,669 vocabulary entries that the domain almost never uses. The output layer, or LM head, scores every one of those entries at every generated token. In Qwen3-4B, that matrix accounts for nearly 10% of parameters. For a domain-specific small model, it is a recurring cost no request can avoid.
Decode is where the bill lands
Inference splits into prefill and decode. Prefill reads the input once and is compute-bound; decode generates token by token and is memory-bandwidth-bound. At each decode step the model reads the KV cache, and in the measured grader a generated token cost 7.4 times a read token. That ratio varies with hardware, batch size and context length, but the direction is consistent: producing tokens is much more expensive than consuming them. Counting total tokens therefore understates the opportunity. A token removed from the model's output matters more than one removed from the input, and each unused vocabulary row still costs the same as a frequent one because the LM head reads the entire matrix every step.
VocabX rebuilds the vocabulary around cost
The VocabX method changes the selection question from frequency to measured cost and time. It rebuilds both the sub-word segmentation and the multi-word tokens under one cost criterion. In the grader, the output layer dropped from about 152,000 to about 65,000 rows, and the reported measured inference cost fell by 77% without changing the base model, training data or recipe.
Prior techniques helped but left gaps. When tested on this problem, the team found two limitations in reference methods:
- Fast vocabulary transfer chooses tokens by frequency, not by per-token decode cost.
- Multi-word tokenization handles whole phrases only.
- Liquid AI's in-place tokenizer expansion continues BPE training from the original vocabulary, but still does not search for repeated spans like full JSON envelopes.
VocabX addresses this by using five miners to find repeated spans across the corpus, each specialized in long words, whole lines, expressions, complete sentences, and punctuation structures such as JSON and tags. The punctuation-structure miner contributed the most on average, and on code it alone accounted for 17.5 points of token reduction. Without all five miners, token reduction on the grader would drop by 55.6 points.
The catch: the paper is still under review, so the 77% figure and other numbers may change. The 7.4x decode/read ratio is specific to the measured grader. But for anyone deploying small language models in portable settings, the message is concrete: tokenizer design and output-layer size are not fixed plumbing. They are levers you can adjust before training, and the unit to optimize is not the token count, but the cost of the tokens you actually generate.