On needle retrieval over Polish scientific documents, Bielik-11B-v3.0-Instruct's accuracy on prompts within its token limit is below 0.70 at all 5 measured lengths from 3,768 to 69,328 characters, ranging from 0.000 to 0.667.

Relation: Accuracy across lengths

metric

Within-window accuracyQuantity measured.

subject

Bielik-11B-v3.0-InstructModel measured.

task

Needle retrieval in scientific documentsTask measured.

language

PolishLanguage of the documents.

threshold

0.70 accuracyAccuracy threshold compared with.

lengths below

5 lengthsMeasured lengths with accuracy below the threshold.

lengths measured

5 lengthsMeasured lengths with prompts within the token limit.

shortest length

3768 charactersShortest measured length.

longest length

69328 charactersLongest measured length within the token limit.

lowest accuracy

0.000 accuracyLowest accuracy across the lengths.

highest accuracy

0.667 accuracyHighest accuracy across the lengths.

Method and evaluation protocol

Graded generations of prompts within the model's token limit, per length.

Dataset

Polish Wikipedia science articles and Polish State Specialization Examination questions from frozen corpora, interleaved and packed to character budgets.Version: unspecified · Access: restricted

Reported results

Accuracy within the token limit by length: 3,768: 0.667 over 30 prompts; 18,839: 0.567 over 30 prompts; 37,678: 0.100 over 30 prompts; 56,517: 0.000 over 30 prompts; 69,328: 0.467 over 30 prompts. Outcomes over these prompts: correct 54, wrong 35, no answer 46, truncated 15. Bielik-PL-11B-v3.0-Instruct on the same task and language: 3,768: 1.000; 18,839: 0.833; 37,678: 1.000; 56,517: 0.967; 69,328: 1.000; 81,384: 1.000; 86,605: 0.967; 106,235: 1.000.

Uncertainty and replication

Point values from one greedy run; 30 prompts per length from 6 needles.

Evidence references

analysis.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c60abf85776b3d991bdbba0e47569c6d944056e7/experiments/E04-effective-context/results/analysis.jsongraded_orig.jsonl · restrictedNot redistributed: ask the Room owner, or rebuild with scripts/03_grade.py at the pinned revisionsmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/c60abf85776b3d991bdbba0e47569c6d944056e7/experiments/E04-effective-context/results/metrics.json

Limitations

The report and Amendment 3 describe the failed answers as replies offering help without answering the question, read from the raw generations, which are not redistributed; replies by kind were not counted in any stored file. Polish documents include examination questions, which some answers follow. The tiny-context gate passed on two smoke prompts, although accuracy at the shortest length is below its 0.90 halt threshold.