In the five packed recovery pretraining streams, each source's realised share of APT4 tokens differs from its target share by at most 0.0249 percentage points.

Relation: Share deviation at most

streams

5 streamsPacked streams compared.

variable

Math and code shareShares compared.

value

0.0249 percentage pointsLargest absolute deviation of any source's share from its target.

Method and evaluation protocol

Realised and target shares from each stream's pack record.

Dataset

Pack records of the five training streams.Version: unspecified · Access: public

Reported results

largest deviation 0.0249 percentage points (mix00: pl 0.800127 against 0.8, en 0.199873 against 0.2; mix10: pl 0.720162 against 0.72, en 0.179925 against 0.18, math 0.049952 against 0.05, code 0.049962 against 0.05; mix30: pl 0.560249 against 0.56, en 0.139959 against 0.14, math 0.149917 against 0.15, code 0.149875 against 0.15; conf00: pl 0.800127 against 0.8, en 0.199873 against 0.2; conf30: pl 0.560249 against 0.56, en 0.139959 against 0.14, math 0.149917 against 0.15, code 0.149875 against 0.15).

Uncertainty and replication

Exact counts.

Evidence references

data_manifests/mix00/mix00.meta.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/data_manifests/mix00/mix00.meta.jsondata_manifests/mix10/mix10.meta.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/data_manifests/mix10/mix10.meta.jsondata_manifests/mix30/mix30.meta.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/data_manifests/mix30/mix30.meta.jsondata_manifests/conf00/conf00.meta.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/data_manifests/conf00/conf00.meta.jsondata_manifests/conf30/conf30.meta.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/data_manifests/conf30/conf30.meta.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/metrics.json

Limitations

The streams are packed in document units, so shares cannot match targets exactly. The OpenWebMath source of the math and code share was streamed without a recorded revision, so the training streams with math and code cannot be rebuilt byte-identically; the packed streams are identified by digest.