On the 100 English-instructed tool-use task cells with at most 8 steps and 700 generated tokens per step, 0.1 of Bielik-PL-11B-v3.0-Instruct's trajectories and 0.36 of Bielik-11B-v3.0-Instruct's end without an accepted FINAL answer.

Relation: Metric comparison

metric

Non-finalization rateQuantity compared.

benchmark

Agentic task suiteTasks measured.

language

EnglishInstruction language of the cells.

setting

ReAct tool loop with a persistent Python sandbox: at most 8 steps, 700 generated tokens per step, a FINAL answer ends a trajectory only in a turn without code; write-and-debug solutions rebuilt from executed code when no solution.py was written; MLX bf16, greedy decodingConditions of the measurement.

subject

Bielik-PL-11B-v3.0-InstructModel after the tokenizer change.

subject value

0.1 proportionRate for the subject.

comparator

Bielik-11B-v3.0-InstructModel before the tokenizer change.

comparator value

0.36 proportionRate for the comparator.

Method and evaluation protocol

Count English-instructed trajectories that end by step-budget exhaustion, context overflow or stalling, divided by the 100 cells.

Dataset

100 synthetic task instances from 20 parametric templates, each with English and Polish instructions.Version: tasks manifest sha256 66029e45c11f07613d698a67498b2c2c29faee4f0acc30bcf667bc3af6306f50 · Access: public

Reported results

Original: 10 budget exhausted, 0 overflow, 26 stalled; APT4 model: 10 budget exhausted, 0 overflow, 0 stalled. Overflow or budget exhaustion alone: 0.1 (original) and 0.1 (APT4 model).

Uncertainty and replication

Counts over 100 cells per model; no interval.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7458cbb2d1925fda6196c42e14dca7ccbc3b295c/experiments/E10-agentic-compounding/results/analysis.jsongraded_summary.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7458cbb2d1925fda6196c42e14dca7ccbc3b295c/experiments/E10-agentic-compounding/results/graded_summary.jsonmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/7458cbb2d1925fda6196c42e14dca7ccbc3b295c/experiments/E10-agentic-compounding/results/metrics.json

Limitations

H3's second clause as designed counts overflow and budget exhaustion only; the analysis reports non-finalization, which also counts stalls. Single greedy run per cell on synthetic tasks; the two models differ by tokenizer, continued pretraining and post-training at once.