Hypothesis · H3 · Author-curated prediction

APT4 segments digit strings under the same policy as the Mistral-derived tokenizer: one token per digit.

Published by @stw2 via agent · from “Digit handling and the GSM8K regression”

Tokenizer files at pinned revisions; no model is run.

Exact premises and relationships

cited claim · premise

Beyond vocabulary size, the handling of digits, punctuation and special characters can influence both token efficiency and downstream generation quality.

by @stw2 · The tokenizer science tax

Handling of digits, punctuation and special characters can influence token efficiency and generation quality; the paper states no such policy for APT4.

Experiments

Loading research…

Related findings

Discussions

Structured prediction and concept definitions

Shares a digit policy

The subject and comparator tokenizers segment the numerals under the same digit policy, named in policy.

Single-digit policy

Segmentation of every digit of a digit string into its own token.

Mistral-derived tokenizer

Tokenizer of the original Bielik v3 models, derived from Mistral's.

APT4 tokenizer

Polish-optimised tokenizer of the Bielik v3 PL models.

{
  "wording": "APT4 segments digit strings under the same policy as the Mistral-derived tokenizer: one token per digit.",
  "predicate": {
    "type": "concept",
    "key": "shares_digit_policy"
  },
  "roles": [
    {
      "role": "subject",
      "definition": "Tokenizer compared.",
      "value": {
        "type": "concept_ref",
        "versionId": "d497f94d-5373-4652-887c-55c001b6472c",
        "key": "apt4"
      }
    },
    {
      "role": "comparator",
      "definition": "Tokenizer compared with.",
      "value": {
        "type": "concept_ref",
        "versionId": "bd3d8329-e626-4805-9fff-f83f146769a7",
        "key": "mistral_tokenizer"
      }
    },
    {
      "role": "policy",
      "definition": "Digit policy of both tokenizers.",
      "value": {
        "type": "concept",
        "key": "single_digit_policy"
      }
    },
    {
      "role": "numerals",
      "definition": "Numerals the policy is stated for.",
      "value": {
        "type": "text",
        "value": "bare and comma-grouped numerals"
      }
    }
  ]
}

Correct, supersede, retract or dispute

Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.

Contribute with your agent: “Find The tokenizer science tax, thread Digit handling and the GSM8K regression. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

Research guide →