Choosing tokenization through training, not compression alone

Does selection by text compression agree with selection by model training on that text?

On 5 July we compared three self-written tokenization alternatives. A tokenizer turns text into model-input units. Preliminary compactness-based selection favoured one alternative; final selection used separate short training runs.

All alternatives received the same ordered text: 6,169 documents and 41,942,946 UTF-8 bytes. The shared held-out set contained 10,051,479 bytes. Runs used a fixed roughly 292-million-parameter proxy architecture and one common seed. Source documents and their order did not change between alternatives.

The budget was matched by source text, not tokens. Equal tokens would let a stronger compressor see more original material. Here token and step counts differed as expected: 11,973,646 and 365; 16,653,321 and 508; 17,236,892 and 526. This matches text exposure, not update count or compute.

The primary measure was bits per source-text byte: normalized prediction error with fixed text-group weights. Lower is better. It allows comparison across segmentations; ordinary per-token loss is not directly comparable for that purpose.

The predefined practical-equivalence threshold was 0.003 bits per byte. A further seed was required only if the two best final scores differed by no more than that threshold. Their 0.0098948263 difference exceeded it, so no additional decision run was used. An intermediate ranking change was not turned into a new post-hoc criterion.

The most compact representation described the same text in fewer tokens. Yet its proxy-training result was worse than both alternatives. The selected representation compressed less but achieved lower normalized prediction error.

The best-to-second gap is small: about 0.43% relative to the second. The 0.003 threshold is a decision rule, not a confidence interval or statistical-significance result. The conclusion is limited to one seed and the stated budget.

This comparison addresses selection under a fixed source-text budget. It does not isolate tokenization at equal compute because update counts differ. Answering that different question requires a separately predefined experiment.

A proxy model and limited text do not establish better semantic correctness, conversation or final large-model quality.

No completed repeated-seed comparisons enter this decision. A practical threshold does not replace variability estimates; universal superiority is not claimed.

This July segmentation comparison is not the August vocabulary-size comparison in another paper. Samples, goals and metrics differ and are not pooled.