Skip to content

feat: Verified Completions - proven core loop, verified benchmark corpus, validated rule pack - #90

Merged
jsam merged 1 commit into
mainfrom
verified-completions-plan
Aug 14, 2026
Merged

feat: Verified Completions - proven core loop, verified benchmark corpus, validated rule pack#90
jsam merged 1 commit into
mainfrom
verified-completions-plan

Conversation

@jsam

@jsam jsam commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

The foundation of Verified Completions (tracking issue #89), shipped as one verified unit: design, engine-validated rule pack, executable benchmark with per-scenario verification, tests at every level, and the docs that explain it.

The proven core loop

Conversation text becomes atomic claims with quoted evidence, claims become facts in a per-conversation knowledge graph, and the consistency rule pack derives findings with proof trees. poc/poc_verify.py runs that loop end to end (extract -> validate -> load -> judge); the scorer and ingestion validator are negative-tested (fabricated quotes get dropped, impossible expectations fail rows).

The rule pack, validated on the real engine (#81)

  • Restructured to one clause per detection relation with unions only in reporting views - works around engine bug Engine: multi-clause rules silently return empty or malformed results when clauses share a derived relation #91 (multi-clause rules silently evaluating empty), external query interface unchanged
  • eq identity helper symmetrized at the base so .why proof queries terminate
  • Ontology seeds hardened by review: shareable identifiers (email, order_number, booking_reference) removed from inverse_functional; attr_domain seeded so the domain rule can fire
  • Enforced by CI: the fixture runs as an e2e snapshot test (examples/iql/42_consistency_pack/), full suite green

The benchmark (corpus v2.1)

poc/corpus.py deterministically generates 1,628 scenarios across 16 families (~100 each, all 48 sub-variants at n>=33) covering value, negation, temporal, spatial, causal, structural, numeric, counting, identity, classification, and instruction corruption, plus 102 controls; 449 scenarios plant the conflict 12+ turns deep. Every sample carries multi-valued category labels, exact conflicting spans, a clean twin, a natural task, and extractor-truth facts. The corpus is documented for consumers in poc/DATASET.md (design principles, scenario anatomy, per-scenario validation, how to run and extend it) and regenerates byte for byte.

Per-scenario verification (poc/verify_each.py, ledger committed): spans/clean-twins/labels 1,628/1,628; engine pass 1,526/1,526 with EXACT finding-kind match - the engine fires precisely the expected kinds and nothing else, on every corrupted scenario, deterministically.

Completed behavioral study (corpus v1, 12 families at n=30, Wilson CIs, double-graded): corruption drops sound outputs from 95% to 87% [83-90] - 38 silent commits to arbitrary values, 4 contradictions written into deliverables; attaching the InputLayer finding restores 98% [96-99]. Corrupted system prompts are the standout: 17% sound alone -> 93% with the finding. Raw data archived in poc/results/archive_v1_full_bench_390.json. The v2.1 LLM re-baseline continues in #95 (resume-until-verified tooling is in this PR; 100/1,628 done).

Also in this unit

Verified on this exact tree

cargo fmt, clippy (-D warnings), full cargo test suite, e2e snapshots (1,142 passed / 0 failed), IQL lint on pack + fixtures, corpus regeneration byte-identical, per-scenario ledger, secret scan clean.

Engine bugs found and filed along the way: #91 (silent empty multi-clause unions), #92 (config/env-var conflicts), #93 (inserts ack while storing nothing after data-dir loss).

Closes #81.

@jsam
jsam force-pushed the verified-completions-plan branch from 301e1b8 to bf5fba4 Compare July 26, 2026 17:49
@jsam jsam changed the title docs: Verified Completions design package and development plan feat: Verified Completions - design package, engine-validated rule pack, CI snapshot test Jul 26, 2026
@jsam
jsam force-pushed the verified-completions-plan branch from bf5fba4 to 3207bb7 Compare July 26, 2026 18:36
@jsam jsam changed the title feat: Verified Completions - design package, engine-validated rule pack, CI snapshot test feat: Verified Completions - proven core loop, executable benchmark, validated rule pack Jul 26, 2026
@jsam
jsam force-pushed the verified-completions-plan branch 6 times, most recently from 182d9a6 to 0cd627f Compare August 1, 2026 13:15
@jsam
jsam force-pushed the verified-completions-plan branch 5 times, most recently from dde3159 to 9b9b7aa Compare August 1, 2026 14:14
@jsam
jsam force-pushed the verified-completions-plan branch from 9b9b7aa to 65ae83e Compare August 14, 2026 06:36
@jsam jsam changed the title feat: Verified Completions - proven core loop, executable benchmark, validated rule pack feat: Verified Completions - proven core loop, verified benchmark corpus, validated rule pack Aug 14, 2026
@jsam
jsam force-pushed the verified-completions-plan branch 3 times, most recently from 9b1430d to d346345 Compare August 14, 2026 06:51
…pus, validated rule pack

One unit: design docs, the consistency rule pack validated on the real
engine, a 1,628-scenario benchmark corpus with per-scenario verification,
the two-regime evaluation harness, CI enforcement, and the docs.

- docs/internals/verified-completions/: RFC, review guide, extraction
  prompts, coverage audit, dataset critique, PLAN.md (issues #81-#95)
- rules/consistency-core.iql: engine-validated; one clause per detection
  relation (works around #91); eq symmetrized so .why terminates;
  ontology seeds hardened by review (shareable identifiers removed,
  attr_domain seeded)
- poc/corpus.py|corpus.json: corpus v2.1 - 16 families, 48 sub-variants
  all n>=33, multi-category labels, clean twins, extractor-truth facts;
  hostile-review hardened (DATASET-CRITIQUE.md); byte-reproducible
- poc/verify_each.py + results/verification_ledger.json: per-scenario
  verification - structural axes 1,628/1,628, engine pass 1,526/1,526
  with exact finding-kind match
- poc/full_bench.py: two regimes, double grading with Opus arbiter,
  tiers, --resume, subscription-OAuth-first auth; completed v1 study
  (n=30/family, Wilson CIs) archived; v2.1 LLM re-baseline -> #95
- examples/iql/42_consistency_pack/: pack enforced by the e2e suite
- scripts/run_snapshot_tests.sh: generated test config, rate limits off,
  credentials-file auth (#92)
- docs/blog/consistency-ontology.md: beginner-facing ontology journey

Verified on this tree: cargo fmt, clippy -D warnings, full cargo test
suite, e2e snapshots 1,142 passed / 0 failed, IQL lint, corpus
regeneration byte-identical, per-scenario ledger, secret scan.
@jsam
jsam force-pushed the verified-completions-plan branch from d346345 to f3af72b Compare August 14, 2026 06:57
@jsam
jsam merged commit 76d0904 into main Aug 14, 2026
11 checks passed
@jsam
jsam deleted the verified-completions-plan branch August 14, 2026 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verified Completions Phase 1: run the consistency rule pack on the real engine

1 participant