On LLMzSzŁ STEM questions, likelihood accuracy of Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer is 0.2333, 0.2933, 0.2967, 0.3100, 0.2833 and 0.2933, and of Qwen2.5-1.5B with its own tokenizer 0.2900, 0.3100, 0.3000, 0.2833, 0.2767 and 0.2800, after 0, 100M, 200M, 300M, 400M and 500M tokens of the same continued pretraining.

Relation: Arm trajectories

metric

Multiple-choice likelihood accuracyQuantity measured.

benchmark

LLMzSzŁ STEM questionsItems scored.

subject

Qwen2.5-1.5B with APT4 by Fast Vocabulary Transfer, continuedArm whose values are subject_values.

subject values

0.2333; 0.2933; 0.2967; 0.3100; 0.2833; 0.2933Accuracy of the subject arm at each milestone.

comparator

Qwen2.5-1.5B continued with its own tokenizerArm whose values are comparator_values.

comparator values

0.2900; 0.3100; 0.3000; 0.2833; 0.2767; 0.2800Accuracy of the comparator arm at each milestone.

milestones

0; 100,139,008; 200,278,016; 300,417,024; 400,031,744; 500,170,752 tokensTraining tokens of each value.

items

300 itemsItems scored per model.

language

PolishLanguage of the items.

Method and evaluation protocol

For each item, the summed log-probability of the answer letter after the question and 'Odpowiedź:'; the highest-scoring letter is the prediction; first items of each set in file order.

Dataset

The first 300 items of E3's LLMzSzŁ STEM questions.Version: unspecified · Access: restricted

Reported results

Arm B 0.23333333333333334, 0.29333333333333333, 0.2966666666666667, 0.31, 0.2833333333333333, 0.29333333333333333; arm A 0.29, 0.31, 0.3, 0.2833333333333333, 0.27666666666666667, 0.28.

Uncertainty and replication

One scoring pass per model; no interval; accuracies lie near the chance level of four-option items.

Evidence references

probe_armA.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/probe_armA.jsonlprobe_armB.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/probe_armB.jsonlanalysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/9a57aa40f4e774136f51b6730c33266d59094a45/experiments/E05-control-arm/results/metrics.json

Limitations

Polish examination items only, so the probe cannot show an English collapse. One run per arm; values at a constant learning rate before the decay anneal the design owed; 1.5B parameters and 0.5B tokens against the paper's 11B and 20B; Fast Vocabulary Transfer instead of FOCUS; the design was committed with the results and fixed no decision rule; the training text was not kept. The time-zero arm-B rows were scored with an earlier tokenizer loader that rebuilt from the checkpoint's own tokenizer.json and were not re-scored; the encodings are probably, not verifiably, equal.