"Expand an LLM Tokenizer Without Starting Over"

A tokenizer is part of a checkpoint's application binary interface, not a text file that can be swapped independently. Liquid AI's public LFM2.5 case shows that a 65,536 to 128,000 vocabulary expansion can preserve a costly pretrained checkpoint, but it still required exact token mapping, 1 trillion adaptation tokens, and hardware-level validation. This guide turns that case into a migration contract, explains the failure ablations, and shows when the performance trade is worth taking.

Reading time: 12 minutes | Word count: approximately 2,400

TL;DR

• Extend the old BPE merge list instead of training an unrelated tokenizer. Preserve old token IDs, special tokens, and exact decomposition into source tokens.
• Copy embeddings for unchanged tokens. Initialize each new row with the mean of the embeddings for its exact source-token decomposition.
• Adapt in two stages. Liquid AI trained only new rows for 600B tokens, then all parameters for 400B tokens on a balanced multilingual mixture.
• Count both sides of performance. The 128K vocabulary reduced token counts by up to 4.0x, but each decode step became 7.4% to 9.4% slower.
• Release tokenizer, checkpoint, config, templates, and converted artifacts as one versioned unit. A partial rollout breaks the checkpoint ABI even if every individual file is valid.

Treat the tokenizer as a checkpoint ABI

The usual mental model is too small: tokenizer.json converts text into integers, and the model consumes those integers. In a pretrained model, each integer already names a learned row in the input embedding and, often, the output LM head. Its meaning also depends on ordered merge rules, normalization, byte fallback, special-token IDs, and the chat template that inserts those tokens.

Changing one artifact can silently change all of the following:

This is why tokenizer expansion is a multi-artifact checkpoint migration. Hugging Face lists the released LFM2.5-8B-A1B with vocab_size: 128000 and tie_word_embeddings: true. Its model card also warns that tokenizer files were updated after release to fix tool-calling behavior in llama.cpp, and that earlier downloads and converted GGUF files needed replacement. That operational incident demonstrates the ABI principle: correct weights plus a stale tokenizer package do not form a correct model.

What Liquid AI actually did

The method starts from a source tokenizer the model producer controls. It does not solve arbitrary replacement with a third-party tokenizer.

1. Continue the old BPE merges

The source was a byte-level BPE tokenizer with a vocabulary of 65,536. Liquid seeded the target tokenizer with the original ordered merge list, froze those merges, and continued BPE training on multilingual data until the target vocabulary reached 128,000.

This construction preserves two useful invariants. Most source tokens retain a one-to-one target identity. Every genuinely new target token can also be decomposed exactly into a sequence of source tokens. In the paper's mapping analysis, 63,151 of 64,400 active source entries transferred one-to-one.

Those numbers should not be simplified into “all old tokens are unchanged.” The migration needs an explicit mapping table for every source entry, including special and inactive cases, and a policy for any exception.

2. Transfer knowledge row by row

For a target token that maps directly to an old token, copy the old embedding row unchanged. For a new token with exact source decomposition (s_1, s_2, ..., s_k), initialize its row as:

E_new(token) = mean(E_old(s_1), E_old(s_2), ..., E_old(s_k))

The initialization is not random and does not require a learned cross-tokenizer alignment. Because LFM2.5 ties input and output embeddings, the same transferred matrix also supplies the LM head. An untied architecture would need separate, consistently mapped input and output matrices.

3. Adapt without moving the old rows first

Liquid then used two controlled stages before its normal mid-training and post-training pipeline:

This is an in-place upgrade, but it is not training-free. The published case consumed 1T adaptation tokens. “Without starting over” means reusing the expensive source checkpoint, not avoiding continued pretraining.

Write the migration contract before training

A reproducible expansion needs a versioned contract, not just a training script. The contract should be reviewable before compute is committed.

source:
  checkpoint_revision: immutable-hash
  tokenizer_revision: immutable-hash
  vocab_size: 65536
  merge_table_digest: sha256:...

target:
  vocab_size: 128000
  preserves_source_merges: true
  old_id_policy: identity_or_documented_exception
  new_row_initializer: exact_source_subtoken_mean

adaptation:
  stage_1: {trainable: new_rows_only, tokens: 600B}
  stage_2: {trainable: all_parameters, tokens: 400B}

release_unit:
  - tokenizer files and model weights
  - config, special-token map, and chat template
  - GGUF, MLX, ONNX, and serving manifests
  - evaluation report and rollback revision

Production IDs and hashes must be read from the pinned source revision. The important property is that the contract makes every coupled artifact and exception visible.

Three preflight checks deserve hard failure behavior:

1. Every source ID has exactly one documented target outcome.
2. Every new token decomposes exactly under the source tokenizer.
3. Tokenizer metadata, model tensor shapes, config, templates, and exported formats agree on vocabulary and special-token semantics.

Read the quality results without overclaiming

Liquid reported an unweighted aggregate over eight benchmarks at four controlled checkpoints:

The 48.3 result cannot be attributed entirely to the tokenizer. Stage 2 adds 400B training tokens, and Liquid explicitly says that the surplus over the source most likely reflects that extra pretraining. The defensible claim is that the expanded model recovered and then exceeded the baseline after continued training, not that doubling vocabulary caused an 8% quality gain.

Global MMLU moved from 41.4 to 43.7 overall. The reported Stage 2 gains were especially large for languages that the source tokenizer fragmented heavily:

These are results from one model family and one data recipe. The paper is a July 2026 preprint and has not received independent peer review. Teams should treat it as a detailed primary-source engineering case, not a universal scaling law.

Learn from the negative ablations

The failures are more useful than a headline aggregate because they identify invariants that a migration must protect.

Letting old embedding rows move can destroy generation

When old rows were trainable during embedding adaptation, HumanEval+ fell from 43.9 to 1.8 and the repetition metric fell from 0.92 to 0.19. Multiple-choice scores could remain stable or even improve while free-form generation collapsed.

Moving rows for already learned tokens changes the input basis seen by the entire frozen network. Small row-level errors then propagate through every layer and back through the tied output matrix. Stage 1 therefore needs a verified gradient mask or parameter partition, plus a post-step assertion that every old row is byte-for-byte unchanged.

A biased Stage 2 mixture can hide catastrophic forgetting

An English-heavy Stage 2 mixture produced another warning:

MMLU looked stable and MMMLU improved, yet math generation and repetition deteriorated sharply. A release gate built only around multiple-choice accuracy would have approved a broken checkpoint. Tokenizer migrations need generative tests, code execution, exact-match math, repetition analysis, and language-specific prompts alongside classification benchmarks.

Keep a real performance ledger

Larger vocabulary reduces the number of decoder steps for fragmented languages. It also enlarges the embedding and LM-head matrix read on every step. On memory-bandwidth-limited edge inference at batch size one, the second cost is material.

Liquid reported these corpus-level compression ratios, defined as source token count divided by expanded token count:

Across an M4 Max CPU, M4 Max GPU, and Snapdragon reference device, the 128K vocabulary made each decoded token 7.4% to 9.4% slower. English, code, and some European languages received little compression, so their per-character decoding could be up to about 9% slower. At 256K vocabulary, the Snapdragon throughput penalty reached as much as 37%, which is why the team stopped at 128K.

The reported 2.2x to 3.7x per-character speedups for underserved languages are synthesized estimates. They combine measured per-token throughput with measured token compression. They are not direct end-to-end application latency measurements.

Your own ledger should separate at least five quantities:

1. Prompt tokens per character by production language and content type.
2. Generated tokens per character on representative responses.
3. Prefill and decode throughput on each target backend and device.
4. Peak memory, package size, and cold-load time.
5. End-to-end task latency, including templating, sampling, tool calls, and network time.

A vocabulary upgrade pays only when compression on the real traffic mix exceeds the larger head's fixed cost. An average across languages can conceal both a strong win for Thai and a regression for an English-only coding workload.

Know where this recipe applies

This approach is a strong candidate when all of the following are true:

• You own the original BPE tokenizer, merge order, normalization, special tokens, and source checkpoint.
• The checkpoint is expensive enough that 1T tokens of adaptation is preferable to full pretraining.
• Important production languages are measurably over-tokenized.
• You can build a balanced continuation corpus and run generative as well as multiple-choice evaluation.
• You control the deployment artifact set and can benchmark actual target hardware.

Choose another method when the target is an unrelated third-party tokenizer, the original merge rules are unavailable, token IDs cannot be kept stable, or there is no continued-pretraining budget. Zero-shot tokenizer transfer methods address a different problem. Starting over may also be cleaner when the source tokenizer's normalization, special-token layout, or language allocation is fundamentally wrong rather than merely too small.

The recipe also needs adaptation for non-BPE tokenizers. “Continue the merges” is a structural advantage of BPE, not a generic instruction for every Unigram or WordPiece implementation.

Evaluation and release checklist

Before training:

• Pin source weights, tokenizer, config, merge table, templates, and special-token map by immutable revision.
• Measure token fertility and characters per token on the actual language and code distribution.
• Verify all old IDs, all exact decompositions, and all exceptions.
• Estimate the enlarged embedding and LM-head memory, bandwidth, and package cost.

After Stage 1:

• Prove old rows did not change, including optimizer state and tied-weight behavior.
• Compare source, zero-swap, and Stage 1 on generation, code, math, repetition, and each target language.
• Inspect token-level log probabilities on old-only prompts and mixed old/new prompts.

After Stage 2:

• Report the continuation-data mixture and keep tokenizer gains separate from extra-pretraining gains.
• Benchmark prefill, per-token decode, per-character decode, memory, and end-to-end latency on every supported backend.
• Test chat templates, stop behavior, structured output, and tool calling.
• Rebuild and checksum every converted artifact. Test clean downloads rather than local caches.
• Ship the tokenizer and checkpoint as one atomic revision with a documented rollback target.

Next action: before allocating training compute, produce the source-to-target token mapping report and a hardware break-even table. If either artifact cannot be reviewed deterministically, the migration is not ready to start.

FAQ

Can I replace a pretrained model's tokenizer and only resize the embeddings?

Usually not safely. Resizing fixes tensor shape, not token identity or learned semantics. The zero-swap aggregate in Liquid's experiment was 38.9 versus 44.7 for the source, even with structured initialization.

Is in-place expansion training-free?

No. Liquid reused the source checkpoint but trained new rows for 600B tokens and the full model for another 400B tokens. It avoids full retraining, not continued pretraining.

Why not train all embedding rows during Stage 1?

Because the old rows define the representation basis the frozen body already understands. In the reported ablation, allowing them to move coincided with HumanEval+ collapsing from 43.9 to 1.8 and severe repetition degradation.

Does fewer tokens always mean faster output?

No. A larger LM head makes every decoder step slower. Languages with high compression can win substantially, while English or code can regress when compression does not cover the 7.4% to 9.4% per-token tax.

Does this recipe work with any third-party tokenizer?

No. Its clean mapping depends on owning the source tokenizer and continuing its exact BPE merges. An unrelated tokenizer needs a different transfer method and a larger alignment risk budget.

References

1. Liquid AI, Tokenizer Expansion: Upgrading a Model's Tokenizer in Place, July 21, 2026.
2. Smith et al., In-Place Tokenizer Expansion for Pre-trained LLMs, arXiv:2607.15232, July 2026 preprint.
3. Liquid AI, LFM2.5-8B-A1B model card, Hugging Face.
4. Liquid AI, LFM2.5-8B-A1B config.json, Hugging Face.
5. Liquid AI, LFM2.5-8B-A1B tokenizer artifact, Hugging Face.
6. Minixhofer, Ponti, and Vulić, Zero-shot Tokenizer Transfer, NeurIPS 2024.