On 200 paired tool-use task cells with English and Polish instructions, at most 8 steps and 2048 generated tokens per step, Bielik-PL-11B-v3.0-Instruct succeeds end to end on 0.7 of cells and Bielik-11B-v3.0-Instruct on 0.725, a gap of 0.025 (95% bootstrap interval -0.055 to 0.105).

Relation: Metric difference with an interval

metric

End-to-end success rateQuantity compared.

benchmark

Agentic task suiteTasks measured.

language

English and Polish instructionsInstruction languages of the cells.

setting

ReAct tool loop with a persistent Python sandbox: at most 8 steps, 2048 generated tokens per step, a FINAL answer also ends a trajectory in a turn with code once earlier code has executed; write-and-debug solutions rebuilt from executed code when no solution.py was written; MLX bf16, greedy decodingConditions of the measurement.

subject

Bielik-PL-11B-v3.0-InstructModel after the tokenizer change.

subject value

0.7 proportionSuccess rate of the subject.

comparator

Bielik-11B-v3.0-InstructModel before the tokenizer change.

comparator value

0.725 proportionSuccess rate of the comparator.

difference

0.025 proportionComparator value minus subject value.

interval low

-0.055 proportionLower bound of the 95% bootstrap interval.

interval high

0.105 proportionUpper bound of the 95% bootstrap interval.

cells

200 task cellsPaired task cells.

Method and evaluation protocol

Grade each trajectory (numeric tolerance or hidden tests), count trajectories without an accepted FINAL answer as failures, and bootstrap the mean paired difference over the cells (2000 resamples, seed 20260707).

Dataset

100 synthetic task instances from 20 parametric templates, each with English and Polish instructions.Version: tasks manifest sha256 66029e45c11f07613d698a67498b2c2c29faee4f0acc30bcf667bc3af6306f50 · Access: public

Reported results

Success 0.725 (original) and 0.7 (APT4 model); gap 0.025 [-0.055, 0.105]; English instructions -0.02 [-0.12, 0.07]; Polish instructions 0.07 [-0.06, 0.19].

Uncertainty and replication

95% percentile bootstrap interval over paired cells, 2000 resamples.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/76ed9b8efec6102935361df4b013764d057e4363/experiments/E10-agentic-compounding/results/analysis.jsongraded_summary.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/76ed9b8efec6102935361df4b013764d057e4363/experiments/E10-agentic-compounding/results/graded_summary.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/76ed9b8efec6102935361df4b013764d057e4363/experiments/E10-agentic-compounding/results/metrics.json

Limitations

Single greedy run per cell on synthetic tasks; the two models differ by tokenizer, continued pretraining and post-training at once.