Untouched Qwen2.5-1.5B has 0.0524 bits per byte on GSM8K problems, the lowest of 10 texts scored; the next lowest is 0.3417 on Python code.

Relation: Lowest among texts

metric

Bits per byteQuantity compared.

subject

Qwen2.5-1.5BModel scored.

evaluation text

GSM8K problemsText with the lowest value.

value

0.0524 bits per byteBits per byte on the evaluation text.

runner up text

English Python codeText with the next lowest value.

runner up value

0.3417 bits per byteBits per byte on the runner-up text.

texts compared

10 textsTexts scored.

scope

first 200 documents of each text; 800 math_clean statementsDocuments scored.

Method and evaluation protocol

Next-token negative log-likelihood in bits summed over non-overlapping 2048-token windows of each document, the first token of each window unscored, divided by the documents' UTF-8 bytes; bf16 on Apple silicon, 128 GB unified memory; arm-B tokenizers loaded through a whitespace canary and the frozen APT4 reference.

Dataset

Ten texts: Polish and English holdouts, seven E1 corpora and math_clean.Version: unspecified · Access: restricted

Reported results

sci_gsm8k 0.0524; sci_python 0.3417; sci_latex 0.6436; sci_arxiv 0.6935; en 0.8294; pl_wiki_sci 1.0930; pl 1.2043; pl_pes 1.4071; math_clean 1.5023; pl_informal 1.6948

Uncertainty and replication

One scoring pass, repeated once with identical values; no interval.

Evidence references

bpb_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/bpb_armA.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

The design and report read the low value as memorisation of GSM8K in Qwen2.5's pretraining data; no decontamination check was run. One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept.