None of the 12 model, language and benchmark cells has a 95% Wilson interval for likelihood multiple-choice accuracy that includes the chance level 0.25; the lowest lower bound is 0.3057.

Relation: Cells including chance

metric

Likelihood multiple-choice accuracyQuantity measured per cell.

count including

0 cellsCells whose interval includes the chance level.

count total

12 cellsCells checked.

chance level

0.25 accuracyAccuracy of uniform guessing among four options.

lowest interval low

0.3057 accuracySmallest lower bound of the cells' 95% Wilson intervals.

cells

Qwen2.5-1.5B and its two 500M-token continued-pretraining checkpoints; English and Polish; translation-paired Belebele and MMLU itemsCells checked.

Method and evaluation protocol

Wilson 95% interval of the length-normalised accuracy of each model, language and benchmark cell over the analysed pairs, compared with 0.25.

Dataset

Translation-paired Belebele and MMLU items after the exclusions.Version: unspecified · Access: public

Reported results

0 of 12 cells include 0.25; lowest lower bound 0.3057.

Uncertainty and replication

Wilson score intervals; no multiplicity adjustment.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/5e8117ffb3f168331ee63f1032cd40a62c6296bd/experiments/E11-cross-language-access/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/5e8117ffb3f168331ee63f1032cd40a62c6296bd/experiments/E11-cross-language-access/results/metrics.json

Limitations

The floor rule is checked on accuracy only; concentrated letter predictions can keep a cell above chance without item knowledge. Letter priors: on Polish MMLU items the checkpoint that kept the original tokenizer predicts A for 4 of 600 items and the APT4 checkpoint predicts B for 437, against 134 to 160 gold items per letter; on Polish Belebele items the APT4 checkpoint predicts B for 360 and the other checkpoint D for 326 of 600; the base model's predictions are the least concentrated (all scored rows, results/metrics.json pred_count_*).