Finding · P236 · Author-curated · Corrected

On the 100 English-instructed tool-use task cells with at most 8 steps and 700 generated tokens per step, Bielik-PL-11B-v3.0-Instruct generates 686.6 tokens per trajectory on average and Bielik-11B-v3.0-Instruct 714, a ratio of 0.9617 (95% bootstrap interval 0.7077 to 1.2411).

Published by @stw2 · 2026-09-14 · Sources, measurements and interpretation are supplied by the author.

Notices · Corrected · this exact version stays citable

L7 P240 On the 100 English-instructed tool-use task cells with at most 8 steps and 2048 generated tokens per step, Bielik-PL-11B… corrects this version
Run 1 used 700 generated tokens per step, which cut the original model's code after its <think> preamble, and a rule that ignored a FINAL answer given in a turn with code even after earlier code had run, which kept the APT4 model's grounded Polish answers from ending its trajectories. Run 2 fixed both (Amendments 2 and 3) and regenerated every trajectory. The English generated-token ratio is 0.7974 [0.5017, 1.0846] instead of 0.9617 [0.7077, 1.2411].
@stw2 via agent · cosmos · 2026-09-14 13:18 UTC · Read link L7 →

Structured assertion

Relation: Metric ratio with an interval

metric: Mean generated tokens per trajectoryQuantity compared.
benchmark: Agentic task suiteTasks measured.
language: EnglishInstruction language of the cells.
setting: ReAct tool loop with a persistent Python sandbox: at most 8 steps, 700 generated tokens per step, a FINAL answer ends a trajectory only in a turn without code; write-and-debug solutions rebuilt from executed code when no solution.py was written; MLX bf16, greedy decodingConditions of the measurement.
subject: Bielik-PL-11B-v3.0-InstructModel after the tokenizer change.
subject value: 686.6 tokensMean for the subject.
comparator: Bielik-11B-v3.0-InstructModel before the tokenizer change.
comparator value: 714 tokensMean for the comparator.
value: 0.9617 ratioSubject mean divided by comparator mean.
interval low: 0.7077 ratioLower bound of the 95% bootstrap interval.
interval high: 1.2411 ratioUpper bound of the 95% bootstrap interval.

Experimental provenance

Method and evaluation protocol: Divide the mean generated tokens per English-instructed trajectory of the APT4 model by the original's and bootstrap the ratio over paired cells (2000 resamples, seed 20260707).
Dataset: 100 synthetic task instances from 20 parametric templates, each with English and Polish instructions.Version: tasks manifest sha256 66029e45c11f07613d698a67498b2c2c29faee4f0acc30bcf667bc3af6306f50 · Access: public
Reported results: Ratio 0.9617 [0.7077, 1.2411]; bootstrap p for a ratio of 1: 0.77, Holm-adjusted 0.77; the original model's turns contain a <think> block in 55 of 100 English cells.
Uncertainty and replication: 95% percentile bootstrap interval over paired cells, 2000 resamples.
Evidence references: analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7458cbb2d1925fda6196c42e14dca7ccbc3b295c/experiments/E10-agentic-compounding/results/analysis.jsongraded_summary.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7458cbb2d1925fda6196c42e14dca7ccbc3b295c/experiments/E10-agentic-compounding/results/graded_summary.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7458cbb2d1925fda6196c42e14dca7ccbc3b295c/experiments/E10-agentic-compounding/results/metrics.json
Limitations: H3 as designed compares prompt-side tokens; the analysis computes generated tokens. Single greedy run per cell on synthetic tasks; the two models differ by tokenizer, continued pretraining and post-training at once.

Paper citations

arXiv:2604.10799v1 →Revision supplied by author

Associated through an explicitly referenced cited claim.

Author’s note

Values from results/analysis.json (H3) and results/graded_summary.json (by_lang.en) at tag e10-run1-results.

Concept definitions

Reuse the defining version and key when the meaning fits your assertion.

Mean generated tokens per trajectory

Mean over trajectories of the tokens a model generates, summed over the trajectory's steps.

Key mean_generated_tokens · version 0ea16ba0-68e0-49a0-934c-ca8946e78a80

Concept JSON

Metric ratio with an interval

The subject and the comparator take subject_value and comparator_value of the metric on the same cells; value is subject_value divided by comparator_value, with a 95% paired bootstrap interval from interval_low to interval_high.

Key metric_ratio_with_interval · version 0ea16ba0-68e0-49a0-934c-ca8946e78a80

Concept JSON

Agentic task suite

100 synthetic task instances, 5 from each of 20 parametric templates in three families (8 CSV analysis, 6 reproduction of a computation from a methods paragraph, 6 write-and-debug), each posed with English and with Polish instructions: 200 task cells per model.

Key agentic_task_suite · version 0ea2c3bc-bc27-4395-9371-ce71dbf2c962

Concept JSON · Defining publication

Bielik-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 model with the original, Mistral-derived tokenizer.

Key bielik_11b_v3_instruct · version ac435442-b907-442a-9d1c-f951fa41d53c

Concept JSON · Defining publication

English

The English language.

Key english · version 86acabda-0237-47be-8e26-a81500c184aa

Concept JSON · Defining publication

Bielik-PL-11B-v3.0-Instruct

The 11B instruction-tuned Bielik v3 PL model with the APT4 tokenizer.

Key bielik_pl_11b_v3_instruct · version 55d90aa4-ac59-4189-bedf-b01cf526fe04

Concept JSON · Defining publication

Exact references

eed1981a-09e9-4aa9-bd57-3db32c7ba4a0

English fertility of the two tokenizers.

64285728-7520-4c5e-9d7a-9dbbe2c874f7

The original model's <think> tag on English retrieval prompts.

Correct, supersede, retract or dispute

Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.