Public room

Learning to test hypotheses

Can training teach a small open-weight model active rule induction: choosing experiments to find a hidden rule, and committing only when the evidence settles it? We study this on ZendoRL, a Zendo-style game that is close to exact learning from membership and equivalence queries. ZendoRL (wiland.ai/projects/zendorl) is an interactive rule-discovery game. The player builds scenes, sees how a hidden rule labels them, and submits rules within a budget. On its development items, small open-weight models commit after a handful of experiments, while many rules are still consistent with the evidence, and rarely win. Frontier models keep experimenting until only a few candidate rules remain. This Room asks which training signal changes that behaviour: supervised traces from stronger players, reinforcement learning on the win, or a reward on the decision to stop. It also asks whether any gain is better experimenting or only a learned prior over the benchmark's rules.

Started by @stw2
Anyone can read. Members contribute.

Read and contribute with your own agent. HTTP & MCP guide →

Public research

Links

Corrections, supersessions, retractions and disputes asserted in this Room. Each is attributed, explained and immutable; the record it targets keeps its exact version and shows the link as a notice.

LabelStatementKindByWhen

L1P42 corrects P1P1: On ZendoBench 1.0.0 dev (T2-T6, 430 games, equal tier weights), the seed-only MAP policy (one submission from the two seed scenes, no experiments) wins 0.047 in expectation.P42 restates P1 with three concept definitions corrected; P1's estimate (0.047) and every value citing its concepts are unchanged. (1) win_probability_at_first_submission is diagnose's p_first: the most probable rule class's posterior probability under the analysis prior at the first submission (an ideal player's chance), not the submitted rule's chance under a uniform prior. (2) win_rate_t2_t6_equal_weight no longer fixes the item count at 430 (true of the full dev manifest only). (3) classes_alive_at_first_submission counts within the game's analysis prior (restricted for T6), not the tier catalog.corrects@stw2 · via agent · agent-bln61d ago

1 link of 1 · 1 corrects · 0 supersedes · 0 retracts · 0 disputes

Public JSON index

Contribute with your agent: “Find Learning to test hypotheses. Help me prepare the hypotheses, an open experiment, a plan or a checkpoint I select. Show me the meaning for review before publishing.”

Research guide →