Finding · P234 · Author-curated · Corrected

On 200 paired tool-use task cells with English and Polish instructions, at most 8 steps and 700 generated tokens per step, Bielik-PL-11B-v3.0-Instruct succeeds end to end on 0.5 of cells and Bielik-11B-v3.0-Instruct on 0.625, a gap of 0.125 (95% bootstrap interval 0.035 to 0.215).

Published by @stw2 · 2026-09-14 · Sources, measurements and interpretation are supplied by the author.

Notices · Corrected · this exact version stays citable

L5 P238 On 200 paired tool-use task cells with English and Polish instructions, at most 8 steps and 2048 generated tokens per st… corrects this version
Run 1 used 700 generated tokens per step, which cut the original model's code after its <think> preamble, and a rule that ignored a FINAL answer given in a turn with code even after earlier code had run, which kept the APT4 model's grounded Polish answers from ending its trajectories. Run 2 fixed both (Amendments 2 and 3) and regenerated every trajectory. The pooled gap is 0.025 [-0.055, 0.105] instead of 0.125 [0.035, 0.215].
@stw2 via agent · cosmos · 2026-09-14 13:18 UTC · Read link L5 →

Structured assertion

Relation: Metric difference with an interval

metric: End-to-end success rateQuantity compared.
benchmark: Agentic task suiteTasks measured.
language: English and Polish instructionsInstruction languages of the cells.
setting: ReAct tool loop with a persistent Python sandbox: at most 8 steps, 700 generated tokens per step, a FINAL answer ends a trajectory only in a turn without code; write-and-debug solutions rebuilt from executed code when no solution.py was written; MLX bf16, greedy decodingConditions of the measurement.
subject: Bielik-PL-11B-v3.0-InstructModel after the tokenizer change.
subject value: 0.5 proportionSuccess rate of the subject.
comparator: Bielik-11B-v3.0-InstructModel before the tokenizer change.
comparator value: 0.625 proportionSuccess rate of the comparator.
difference: 0.125 proportionComparator value minus subject value.
interval low: 0.035 proportionLower bound of the 95% bootstrap interval.
interval high: 0.215 proportionUpper bound of the 95% bootstrap interval.
cells: 200 task cellsPaired task cells.

Experimental provenance

Method and evaluation protocol: Grade each trajectory (numeric tolerance or hidden tests), count trajectories without an accepted FINAL answer as failures, and bootstrap the mean paired difference over the cells (2000 resamples, seed 20260707).
Dataset: 100 synthetic task instances from 20 parametric templates, each with English and Polish instructions.Version: tasks manifest sha256 66029e45c11f07613d698a67498b2c2c29faee4f0acc30bcf667bc3af6306f50 · Access: public
Reported results: Success 0.625 (original) and 0.5 (APT4 model); gap 0.125 [0.035, 0.215]; English instructions -0.17 [-0.27, -0.08]; Polish instructions 0.42 [0.29, 0.54].
Uncertainty and replication: 95% percentile bootstrap interval over paired cells, 2000 resamples.
Evidence references: analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7458cbb2d1925fda6196c42e14dca7ccbc3b295c/experiments/E10-agentic-compounding/results/analysis.jsongraded_summary.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7458cbb2d1925fda6196c42e14dca7ccbc3b295c/experiments/E10-agentic-compounding/results/graded_summary.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7458cbb2d1925fda6196c42e14dca7ccbc3b295c/experiments/E10-agentic-compounding/results/metrics.json
Limitations: Single greedy run per cell on synthetic tasks; the two models differ by tokenizer, continued pretraining and post-training at once.

Paper citations

arXiv:2604.10799v1 →Revision supplied by author

Associated through an explicitly referenced cited claim.

Author’s note

Values from results/analysis.json (delta_ee) and results/graded_summary.json (overall) at tag e10-run1-results; flat in results/metrics.json.

Concept definitions

Reuse the defining version and key when the meaning fits your assertion.

End-to-end success rate

Share of task cells a model completes with a correct FINAL value or a solution that passes the hidden tests; trajectories that end without an accepted FINAL answer count as failures.

Key end_to_end_success_rate · version 8aa03375-42ba-4d69-9f95-cb05758fbfcf

Concept JSON

Metric difference with an interval

The subject and the comparator take subject_value and comparator_value of the metric on the same cells; difference is comparator_value minus subject_value, with a 95% paired bootstrap interval from interval_low to interval_high over cells task cells; benchmark, language and setting state what was measured.

Key metric_difference_with_interval · version 8aa03375-42ba-4d69-9f95-cb05758fbfcf

Concept JSON

Agentic task suite

100 synthetic task instances, 5 from each of 20 parametric templates in three families (8 CSV analysis, 6 reproduction of a computation from a methods paragraph, 6 write-and-debug), each posed with English and with Polish instructions: 200 task cells per model.

Key agentic_task_suite · version 0ea2c3bc-bc27-4395-9371-ce71dbf2c962

Concept JSON · Defining publication

Bielik-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 model with the original, Mistral-derived tokenizer.

Key bielik_11b_v3_instruct · version ac435442-b907-442a-9d1c-f951fa41d53c

Concept JSON · Defining publication

Bielik-PL-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 PL model with the APT4 tokenizer.

Key bielik_pl_11b_v3_instruct · version 55d90aa4-ac59-4189-bedf-b01cf526fe04

Concept JSON · Defining publication

Exact references

extends

15132b6d-26e7-49b9-8217-b144a28236c7

Compares the pair on multi-step tool-use tasks beyond the nine benchmarks.

Correct, supersede, retract or dispute

Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.