Training the Polish-only 32k tokenizer a second time from the same input gives identical model content once the stored output path is cleared and a byte-identical tokenizer.json; the raw model files differ.

Relation: Retrain identity

subject

Polish-only 32k tokenizerTokenizer trained twice.

content identical

trueModel content identical with the output path cleared.

raw identical

falseRaw model files byte-identical.

converted identical

trueConverted tokenizer.json byte-identical.

Method and evaluation protocol

Retrain the arm from its frozen input into a second output path, compare the serialised models with trainer_spec.model_prefix cleared, the raw files and the converted tokenizer.json.

Dataset

The Polish-only arm's 1 GiB training input.Version: unspecified · Access: restricted

Reported results

Model content identical true; raw bytes identical false; tokenizer.json identical true. With trainer_spec.input also cleared, the retrained model equals the shipped Polish-only spm.model.

Uncertainty and replication

One retrain of one arm.

Evidence references

tokenizers_e9_manifest.json · unavailableArchived private repository, experiments/E9/results/tokenizers_e9_manifest.json at commit e4f0cad; the public copy lists the five scrubbed spm.model digestsmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c5299916db7f792a209ff2944e66c673c3ebb817/experiments/E09-vocabulary-allocation/results/metrics.jsonTokenizer training attempt outcome · publicDelivered outcome of the attempt: exit 1 after all five arms were built and passed G1, G4 and G5; G3 model_identical=False json_identical=True

Limitations

The first comparison, on raw bytes, failed; the rule was amended afterwards and the passing comparison comes from an unlogged re-run. The byte-identical atlas re-run the gate also requires has no record.