On the tool-use tasks with at most 8 steps and 2048 generated tokens per step, the end-to-end success gap of Bielik-11B-v3.0-Instruct over Bielik-PL-11B-v3.0-Instruct is -0.02 with English instructions and 0.07 with Polish instructions, a difference of -0.09 (95% bootstrap interval -0.24 to 0.07).

Relation: Difference in differences

metric

End-to-end success rateQuantity whose difference is taken.

benchmark

Agentic task suiteTasks measured.

setting

ReAct tool loop with a persistent Python sandbox: at most 8 steps, 2048 generated tokens per step, a FINAL answer also ends a trajectory in a turn with code once earlier code has executed; write-and-debug solutions rebuilt from executed code when no solution.py was written; MLX bf16, greedy decodingConditions of the measurement.

subject

Bielik-PL-11B-v3.0-InstructModel after the tokenizer change.

comparator

Bielik-11B-v3.0-InstructModel before the tokenizer change.

english difference

-0.02 proportionComparator minus subject on English-instructed cells.

polish difference

0.07 proportionComparator minus subject on Polish-instructed cells.

value

-0.09 proportionEnglish difference minus Polish difference.

interval low

-0.24 proportionLower bound of the 95% bootstrap interval.

interval high

0.07 proportionUpper bound of the 95% bootstrap interval.

Method and evaluation protocol

Take the paired success difference on the 100 English-instructed and the 100 Polish-instructed cells, subtract, and bootstrap the difference (2000 resamples, seed 20260707); Holm adjustment over H2 and H3.

Dataset

100 synthetic task instances from 20 parametric templates, each with English and Polish instructions.Version: tasks manifest sha256 66029e45c11f07613d698a67498b2c2c29faee4f0acc30bcf667bc3af6306f50 · Access: public

Reported results

DD -0.09 [-0.24, 0.07]; bootstrap p 0.291, Holm-adjusted 0.328.

Uncertainty and replication

95% percentile bootstrap interval, 2000 resamples; p is twice the smaller share of resamples on either side of 0.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/76ed9b8efec6102935361df4b013764d057e4363/experiments/E10-agentic-compounding/results/analysis.jsongraded_summary.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/76ed9b8efec6102935361df4b013764d057e4363/experiments/E10-agentic-compounding/results/graded_summary.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/76ed9b8efec6102935361df4b013764d057e4363/experiments/E10-agentic-compounding/results/metrics.json

Limitations

The bootstrap resamples English and Polish cells independently, not by instance. Single greedy run per cell on synthetic tasks; the two models differ by tokenizer, continued pretraining and post-training at once.