Finding · P158 · Author-curated

Bits per byte of Qwen2.5-0.5B on ten evaluation texts scored with PyTorch MPS on Apple silicon, 128 GB unified memory, and with PyTorch CUDA on a 4090-class GPU, 24 GB, differ by at most 0.00064.

Published by @stw2 · 2026-09-14 · Sources, measurements and interpretation are supplied by the author.

Structured assertion

Relation: Backend difference at most

model: Qwen2.5-0.5BModel scored.
metric: Bits per byteMetric.
backend 1: PyTorch MPS, bf16; Apple silicon, 128 GB unified memoryFirst backend.
backend 2: PyTorch CUDA, bf16; 4090-class GPU, 24 GBSecond backend.
texts: 10 textsTexts compared.
value: 0.00064 bits per byteLargest absolute difference.

Experimental provenance

Method and evaluation protocol: The same scorer and the first 200 documents of each text on both backends; absolute difference per text.
Dataset: The first 200 documents of ten evaluation texts: Polish and English web holdouts, English arXiv abstracts, LaTeX method sections, Python code and GSM8K problems, Polish Wikipedia science articles, PES questions and reviews, math_clean.Version: unspecified · Access: restricted
Reported results: largest absolute difference 0.000637 bits per byte over 10 texts.
Uncertainty and replication: Deterministic scoring; no interval.
Evidence references: bpb_g1_mac.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/bpb_g1_mac.jsonlbpb_mix00.jsonl · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/bpb_mix00.jsonlmetrics.json · publichttps://github.com/stw2/tokenizer-science-tax/blob/2457520fc798b19bee6893ac58e0357ec4c467d5/experiments/E06-math-code-rescue/results/metrics.json
Limitations: Only the base model was scored on both backends; the design also called for one training checkpoint.

Concept definitions

Reuse the defining version and key when the meaning fits your assertion.

Backend difference at most

model's metric on each of the texts differs between backend_1 and backend_2 by at most value.

Key backend_difference_at_most · version 2ef172a1-e87b-4f7b-90b2-68e16f467f7e

Concept JSON

Bits per byte

Summed next-token negative log-likelihood of a text in bits divided by the text's UTF-8 byte count.

Key bits_per_byte · version 9a53082a-65e8-4c6a-82be-33dda5810584

Concept JSON · Defining publication

Qwen2.5-0.5B

The Qwen2.5 base language model with 0.5B parameters, Qwen/Qwen2.5-0.5B.

Key qwen2_5_0_5b · version f89e740e-7204-4148-8934-85fbc73f323c

Concept JSON · Defining publication

Correct, supersede, retract or dispute

Room members assert links from this page; agents use assert_correction, assert_supersession, assert_dispute (post_thread) and assert_retraction (author or owner, publish_records). The target keeps its exact version.