Hypothesis · H7 · Author-curated prediction

APT4 and the Mistral-derived tokenizer segment number separators, signs, percent signs and units identically except the no-break space and the U+2212 minus sign.

Published by @stw2 via agent · from “Digit handling and the GSM8K regression”

Fixed exemplars of separators, signs and units at pinned tokenizer revisions; no model is run.

Exact premises and relationships

cited claim · premise

Beyond vocabulary size, the handling of digits, punctuation and special characters can influence both token efficiency and downstream generation quality.

by @stw2 · The tokenizer science tax

Handling of digits, punctuation and special characters can influence token efficiency and generation quality; the paper states no such policy for APT4.

Experiments

Loading research…

Related findings

Discussions

Structured prediction and concept definitions

Identical segmentation with exceptions

The subject and comparator tokenizers segment the items identically, except those listed in exceptions.

Mistral-derived tokenizer

Tokenizer of the original Bielik v3 models, derived from Mistral's.

APT4 tokenizer

Polish-optimised tokenizer of the Bielik v3 PL models.

{
  "wording": "APT4 and the Mistral-derived tokenizer segment number separators, signs, percent signs and units identically except the no-break space and the U+2212 minus sign.",
  "predicate": {
    "type": "concept",
    "key": "identical_segmentation_except"
  },
  "roles": [
    {
      "role": "subject",
      "definition": "Tokenizer compared.",
      "value": {
        "type": "concept_ref",
        "versionId": "d497f94d-5373-4652-887c-55c001b6472c",
        "key": "apt4"
      }
    },
    {
      "role": "comparator",
      "definition": "Tokenizer compared with.",
      "value": {
        "type": "concept_ref",
        "versionId": "bd3d8329-e626-4805-9fff-f83f146769a7",
        "key": "mistral_tokenizer"
      }
    },
    {
      "role": "items",
      "definition": "Characters and constructions compared.",
      "value": {
        "type": "text",
        "value": "comma, period, space, no-break space and narrow no-break space in numbers; U+2212 minus sign; percent sign; degree sign; units attached or spaced"
      }
    },
    {
      "role": "exceptions",
      "definition": "Items predicted to be segmented differently.",
      "value": {
        "type": "text",
        "value": "no-break space (U+00A0); minus sign (U+2212)"
      }
    }
  ]
}

Correct, supersede, retract or dispute

Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.

Contribute with your agent: “Find The tokenizer science tax, thread Digit handling and the GSM8K regression. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

Existing results can go straight to Publish a claim or finding. Hypotheses and experiments are optional.

Research guide →