IOLing Research Report

Dataset And Audit

Matched Experiments

Verified Microsteps

These 40 source-disjoint intermediate tasks were transcribed from six rendered official solution pages. The scorer requires a keyed JSON answer and only reviewed semantic equivalences.

Verified Composition

Target-specific cards contain manually checked components but no complete accepted answer unit. They are teacher-only diagnostics, never raw evaluation inputs. Exact traces were manually reviewed before being counted as valid reasoning.

Scaffold Diagnostic

The same six held-out focus records were sampled eight times per method. These prompts expose checked facts or answers and measure whether a useful student-like trace can be elicited; they are not raw-task scores.

Answer-Conditioned Trace Audit

Exact final answers were common because the target answer was supplied. Manual review still rejected every newly sampled v14 trace because at least one derivation claim contradicted the examples, misquoted a rule, or was unsupported.

Conclusions

• The broad 555-record corpus has complete source-level issue coverage, not complete correction: all 130 source problems still have unresolved findings and all 555 records are marked not ready.
• The current deterministic v14 set has 123 train and 24 validation tasks from 15 and 3 source problems. Every task and label was checked against rendered official problem and solution pages, and the source split is disjoint.
• The v15 manual expansion adds 23 directly transcribed Lakhota and Catalan tasks, bringing the checked training set to 146 tasks from 17 source problems. A second visual read corrected two Lakhota transcription errors before training.
• The v16 manual expansion adds 9 Micmac transcription and orthography tasks from the rendered 2008 problem and solution pages, bringing the checked training set to 155 tasks from 18 source problems. The PDF's visual schwa and text-layer @ encoding are represented explicitly rather than silently conflated.
• The matched v16 breadth SFT update regressed microstep coverage from 50/320 and 14/40 in v15 to 47/320 and 13/40, lost the prior Arammba-36 positive, and remained 0/192 on raw held-out answers. It is not the selected checkpoint.
• On the nine new Micmac source-seen tasks, v16 produced only 2/72 exact finals; both were manually rejected as reasoning traces, confirming that exact final reward is still not a reliable process label even on the corrected expansion.
• The clean v11 prompt preserves all v10 targets and labels while removing token-budget instructions that distracted generation.
• Several earlier expanded-SFT runs inherited a ten-epoch checkpoint that had already memorized the original training tasks. They cannot establish clean transfer from Qwen3-4B.
• Fresh-base two-epoch SFT produced four exact train samples and zero validation exact samples; manual review rejected all four rationales as unsupported guesses.
• Adding official source-level rules to fresh-base SFT reduced truncation but still produced zero held-out exact answers. Its two source-seen exact outputs also had invalid reasoning.
• Adding eight directly checked Abui tasks produced zero held-out exact answers. The only exact result in a 12-source training probe had an invented rationale, and the model failed all eight samples on the new source-seen Abui task.
• Adding 35 checked Kimbundu, Hakhun, and Terena tasks produced zero exact validation answers after matched five-epoch rule-rich SFT. The 15-source source-seen probe found only one exact box in 120 samples, with an invalid rationale.
• Before those 35 tasks entered training, the base model found seven exact boxes and the 12-source checkpoint found two across 280 samples each. Manual review rejected all nine raw rationales because they contained false or contradictory claims.
• Giving Qwen3-4B checked source rules or target-specific derivations made all 104 scaffolded final boxes exact, but strict manual review accepted zero raw traces. Useful distillation data still requires manual correction.
• A two-epoch, 5e-6 LoRA bootstrap on all 123 concise checked traces improved validation format from 41.1% to 91.1% and reduced truncation from 59.4% to 8.9%, but validation exact accuracy remained zero.
• The same concise checkpoint produced 12 exact boxes across four source-seen tasks in a 984-sample train probe. Manual review accepted zero as reasoning traces: most were answer-only, and the long traces contained false intermediate claims.
• Qwen3-4B Thinking used the full 4K budget on every validation sample. At 8K, 31/32 focused samples still truncated; manual review found unstable rule assignments and repetition rather than near-complete correct solutions.
• Five-epoch expanded SFT produced sparse source-seen exact answers; 20 epochs memorized all 61 train tasks while validation remained zero.
• Qwen3.6-35B-A3B found one of 24 held-out tasks across eight 8K samples, but severe repetition and truncation make its raw traces unsuitable for SFT.
• Thinking-channel-aware scoring ignores draft boxes before the last closing thinking tag. Re-scoring confirmed that the remaining zeroes are model errors, not alternate-format false negatives.
• Answer conditioning made 90/96 final boxes exact, and adding official rules made 89/96 exact, but manual review accepted 0/24 first traces in each condition. The model usually copied the target while inventing the derivation.
• Adding 19 checked Ubykh and Koryak tasks did not produce a held-out exact answer after matched five-epoch SFT. Only one of six source-seen exact outputs had an acceptable rationale.
• Only 2 of 13 strict-correct post-RL samples had manually acceptable reasoning. Exact final-answer reward must not be used as a rationale-quality label.
• Accuracy-only RL and verifier-grounded RLSD remain closed: the selected-trace checkpoint has no source-disjoint exact reward and no manually valid exact reasoning trace.
• Direct on-policy distillation is not blocked by zero rewards. The first rule-hinted forward-KL pilot trained on raw student prefixes, but it produced zero held-out exact answers and zero valid exact rationales; stronger target-specific facts require a small leakage-audited test.
• A second target-facts OPD diagnostic used mean per-token KL but covered only 14 trajectories from four sources. It also produced zero held-out exact answers and zero valid exact rationales, so neither direct-KL objective should be scaled as implemented.
• A corrected four-condition reference audit found low alignment (cosine 0.213) between question-conditioned and reference-only effects, so reference subtraction was not justified.
• Expanded checked-derivation OPD covered 61 raw prefixes, 12 sources, 15,499 tokens, and 16 updates. It still produced zero held-out exact answers; all eight source-probe exact outputs had invalid or absent reasoning. Direct KL is closed pending a different supervised intermediate representation.
• A matched prefix-compatibility audit found 0.869 mean top-16 overlap and 99.4% privileged-teacher acceptance of sampled student tokens. Prune-OPD weighting would retain all 15,499 tokens, so prefix drift does not explain the failed same-model update; the teacher is insufficiently corrective.
• Structured multitask SFT separated reusable rules from target application, but all 48 held-out generated analyses were manually rejected and both raw and two-stage evaluation remained at zero exact answers. Formatting supervision alone does not make the intermediate claims true.
• Qwen3-30B was materially different from the 4B student, but manual review accepted only one of six privileged prose traces. Sparse top-20 distillation retained 99.846% of teacher mass over 7,808 tokens and still yielded zero held-out exact answers; all 174 visible finals were wrong.
• Forty source-disjoint validation microsteps and 100 training microsteps were manually checked. The reviewed scorer found 57/320 base samples correct across 9/40 tasks; sentence-wrapped mappings and symbolic order variants are handled without general substring or fuzzy matching.
• Balanced microstep SFT expanded held-out pass@8 coverage from 9/40 to 14/40 and improved raw completion to 98.4%, but raw exact accuracy remained 0/192. All 189 visible finals were manually checked and wrong.
• One exact-reward OpenRLHF microstep episode had final rollout reward 0.21875 with no reward-fetch failure. Matched held-out microsteps stayed 14/40 and raw finals stayed 0/192, so a second episode is closed.
• Target-specific verified cards that exposed no complete answer unit elicited 23/192 correct held-out finals across 9/24 records. Manual review accepted 13/23 rationales, establishing guided support but also showing that exact answers remain an unsafe rationale label.
• Manual review found four scorer false positives from unresolved Nahuatl metanotation M and six false negatives from exact reviewed formatting or paraphrase variants. Unit-level case overrides and explicit aliases now handle these without fuzzy matching.
• A 32-record, 14-source composition OPD pass used 3,425 raw-prefix tokens and target cards, but raw held-out accuracy stayed 0/192 and microstep pass@8 fell from 14/40 to 13/40. A second pass is closed.
• Guided sampling yielded 44 exact boxes, but manual review accepted only 18 explanations across 12 records. Answer-exposed reconstruction added 10 accepted records, and 10 remaining traces were written manually; all 32 SFT prompts remained raw.
• Two-epoch continuation SFT on those 32 verified traces left raw accuracy at 0/192 and microstep sample accuracy at 47/320 while pass@8 fell from 14/40 to 13/40. All 192 raw outputs and all 47 microstep positives were manually reviewed. This checkpoint does not replace the balanced baseline.
• One conservative breadth SFT update from the balanced microstep checkpoint raised microstep sample accuracy from 47/320 to 50/320 without changing task pass@8 (14/40). All 50 positives were genuine, but the update produced 0/192 raw held-out exact answers, so it is retained as a diagnostic rather than promoted.
• Further work should target explicitly scored intermediate composition decisions or materially broader checked source coverage; scaling the same 32-row continuation update is closed.