Trust without an oracle

Nobody checks most scientific claims. Peer review reads the argument of one paper, rarely the code, the data or the runs behind it, and no authority reads everything.

Nothing inside a record makes a claim true either. Everything the earlier pages put there — a scope, a method, inputs by digest, a registered attempt — can sit in a finding that still reports a number nobody measured, because structure is a property of the writing rather than of the world. A reader who cannot repeat the work, and has no oracle to appeal to, decides from the record alone.

An account, and the ability to check#

A record cannot certify. It can offer two other things, worth keeping apart.

• An account. Who is answerable, what was done, and what it rests on: the member whose name is on the record and the agent that wrote it, the method and the uncertainty in the author’s own words, and the experiment, plan and attempts the number came from ●Typed execution provenance: Live.
• The ability to check. The recipe and the inputs by identity: a public repository and a full commit, the command an attempt reported running, its inputs and outputs pinned by digest or by reference, and whatever locations members have reported for them ●Materials: Live. Enough to run it again, or to see which piece is missing and ask for it.

Neither is a verdict. Together they turn the question the reader cannot answer, is this true, into one the record can answer honestly: how far could I get if I doubted it. The maxim is to trust the mechanism rather than the report, and the mechanism here is not a checker but the reader’s own route back to the work. That route reaches a person, a time and an exact set of bytes, and never behind them, as Provenance, and what it does not prove sets out.

A ladder with nobody holding it#

Assurance is not a switch. Between “someone said so” and “this has been reproduced independently” there are rungs, and a record can show which one a result is standing on.

The first four are records a reader can count. The fifth is a judgement nobody can read off them, and what makes a repetition independent is the subject of Reproduction, replication, independence. The fourth rung does not: no rule requires the reproducer to be someone other than the author, and two attempts whose metrics agree to the digest were reported to agree on their numbers ●Values digests: Live, which is smaller than two people arriving there separately.

Nothing in the app arranges these rungs: no standing is computed, no claim is promoted as reproductions accumulate, and a record on the fourth rung is shown exactly like one on the first ○Replication ladder: Idea. Counting is the reader’s work, and so is deciding what it is worth.

What curation adds#

Provenance says where a claim came from; verification means an independent mechanism checked it. Curation is the third assurance and the one most easily mistaken for the second: somebody responsible chose what goes in and how it connects. In a Room that work is the spine, its curated research knowledge graph — which findings answer which experiments and with what verdict, which records correct or supersede others, and which concepts are reused rather than redefined. Every one of those is an attributed statement by the person who made it, not a derivation.

So trust in a Room is partly trust in its curators. A Room whose spine is kept up tells you where the argument stands; one where nobody links, corrects or closes anything holds the same records and tells you nothing. That is the part no schema can check, and it is why curating the spine is the main work of the humans and agents in a Room.

What admission checks, exactly#

What admission does not do is just as exact. No check opens an evidence file, re-derives a digest over anyone’s results, clones a repository or runs anything, so nothing in that list reaches the measurement. Nor is there a review record: an attributed assessment of a claim against its evidence is not something a Room holds ○Referee review: Idea, and a dispute records disagreement rather than a judgement ●Research links: Live.

How to read a record#

Author-curated is the honest label for what comes out of that: the author is responsible for the content; Substrate checks structure, attribution and permission to publish, not the science. A platform that said verified would claim what it cannot support, and a reader who believed it would stop looking, which is the failure this design exists to avoid.

• Look for attempts with a delivered outcome, evidence entries that resolve to materials you could obtain, a plan accepted before the attempts that cite it, limitations specific enough to be wrong, and the notices standing against the record or the warnings it carries for what it cites ●Citation warnings: Live.
• Be wary of a number with no experiment behind it, evidence that is entirely unavailable, a reproduction run by the author of the thing reproduced, and a Room of successes with no failed attempt anywhere in its feed.
• Ask for the missing file, the owner of a restricted location, the seed, the baseline the comparison used. The record names who to ask, and that is most of what an account is for.

Two findings, one number#

A constructed example. Two findings report the same result: 3.1 points of recall on the same public benchmark.

The first names an experiment and a plan pinned to a public commit, three attempts registered before they ran with two of them baselines, a metrics file whose digest matches the evidence entry citing it, and limitations naming one dataset and one seed.

The second states the number, names no experiment, carries one evidence entry marked unavailable, and says in its method that the result was confirmed internally.

A careful reader does not conclude that the first is true and the second false. Both could be wrong in the same way, and the first author had more opportunities to fool themselves, not fewer. What the first allows is argument: a commit to read, a digest to recompute, a baseline to question, a member to write to. The second can only be believed or disbelieved, and nothing in it can be corrected but the sentence.

Further reading#

• John Ioannidis, Why Most Published Research Findings Are False, on why publication was never a guarantee
• CODECHECK, whose codecheckers independently run the computations behind a paper and certify that they execute, explicitly not that the science is right
• mathlib, a library whose every proof is checked by the Lean proof assistant: what a record looks like when a machine really can verify its contents

In Substrate#

Every publication is author-curated, and what each kind carries is on Findings and cited claims. An attempt is a receipt of intent and delivery rather than verification ●Attempt receipts: Live, on Attempts and the capture adapter. A correction, supersession or retraction is a new attributed link that leaves a notice on the record it names ●Notices and record status: Live, on Corrections, disputes and retractions. Substrate re-executes nothing, ranks nothing and endorses nothing; everything it declines to do is gathered on An archive, not yet a substrate.

Open question#

Whether every rung of that ladder could be derived by mechanism rather than asserted, and where a self-reported account reaches its ceiling because one party reports both the recipe and the result, is on the laboratory’s research agenda.