On the tool-use tasks with at most 8 steps and 2048 generated tokens per step, 0 of the 3 task families have a 95% bootstrap interval of the end-to-end success gap between Bielik-11B-v3.0-Instruct and Bielik-PL-11B-v3.0-Instruct that excludes 0.

Relation: Intervals excluding zero

metric

End-to-end success rateQuantity whose difference is taken.

benchmark

Agentic task suiteTasks measured.

setting

ReAct tool loop with a persistent Python sandbox: at most 8 steps, 2048 generated tokens per step, a FINAL answer also ends a trajectory in a turn with code once earlier code has executed; write-and-debug solutions rebuilt from executed code when no solution.py was written; MLX bf16, greedy decodingConditions of the measurement.

subject

Bielik-PL-11B-v3.0-InstructModel after the tokenizer change.

comparator

Bielik-11B-v3.0-InstructModel before the tokenizer change.

groups

CSV analysis; methods-paragraph reproduction; write-and-debugGroups of cells compared.

count excluding zero

0 familiesGroups whose interval excludes 0.

count total

3 familiesGroups compared.

Method and evaluation protocol

Bootstrap the paired success difference within each task family (2000 resamples, seed 20260707).

Dataset

100 synthetic task instances from 20 parametric templates, each with English and Polish instructions.Version: tasks manifest sha256 66029e45c11f07613d698a67498b2c2c29faee4f0acc30bcf667bc3af6306f50 · Access: public

Reported results

CSV analysis 0.0375 [-0.0125, 0.0875] over 80 cells; methods-paragraph reproduction -0.05 [-0.2, 0.1] over 60 cells; write-and-debug 0.0833 [-0.1167, 0.2833] over 60 cells.

Uncertainty and replication

95% percentile bootstrap intervals, 2000 resamples each; no multiplicity adjustment.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/76ed9b8efec6102935361df4b013764d057e4363/experiments/E10-agentic-compounding/results/analysis.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/76ed9b8efec6102935361df4b013764d057e4363/experiments/E10-agentic-compounding/results/metrics.json

Limitations

Single greedy run per cell on synthetic tasks; the two models differ by tokenizer, continued pretraining and post-training at once.