feat: Verified Completions - proven core loop, verified benchmark corpus, validated rule pack - #90
Merged
Merged
Conversation
7 tasks
jsam
force-pushed
the
verified-completions-plan
branch
from
July 26, 2026 17:49
301e1b8 to
bf5fba4
Compare
jsam
force-pushed
the
verified-completions-plan
branch
from
July 26, 2026 18:36
bf5fba4 to
3207bb7
Compare
8 tasks
jsam
force-pushed
the
verified-completions-plan
branch
6 times, most recently
from
August 1, 2026 13:15
182d9a6 to
0cd627f
Compare
16 tasks
jsam
force-pushed
the
verified-completions-plan
branch
5 times, most recently
from
August 1, 2026 14:14
dde3159 to
9b9b7aa
Compare
jsam
force-pushed
the
verified-completions-plan
branch
from
August 14, 2026 06:36
9b9b7aa to
65ae83e
Compare
jsam
force-pushed
the
verified-completions-plan
branch
3 times, most recently
from
August 14, 2026 06:51
9b1430d to
d346345
Compare
…pus, validated rule pack One unit: design docs, the consistency rule pack validated on the real engine, a 1,628-scenario benchmark corpus with per-scenario verification, the two-regime evaluation harness, CI enforcement, and the docs. - docs/internals/verified-completions/: RFC, review guide, extraction prompts, coverage audit, dataset critique, PLAN.md (issues #81-#95) - rules/consistency-core.iql: engine-validated; one clause per detection relation (works around #91); eq symmetrized so .why terminates; ontology seeds hardened by review (shareable identifiers removed, attr_domain seeded) - poc/corpus.py|corpus.json: corpus v2.1 - 16 families, 48 sub-variants all n>=33, multi-category labels, clean twins, extractor-truth facts; hostile-review hardened (DATASET-CRITIQUE.md); byte-reproducible - poc/verify_each.py + results/verification_ledger.json: per-scenario verification - structural axes 1,628/1,628, engine pass 1,526/1,526 with exact finding-kind match - poc/full_bench.py: two regimes, double grading with Opus arbiter, tiers, --resume, subscription-OAuth-first auth; completed v1 study (n=30/family, Wilson CIs) archived; v2.1 LLM re-baseline -> #95 - examples/iql/42_consistency_pack/: pack enforced by the e2e suite - scripts/run_snapshot_tests.sh: generated test config, rate limits off, credentials-file auth (#92) - docs/blog/consistency-ontology.md: beginner-facing ontology journey Verified on this tree: cargo fmt, clippy -D warnings, full cargo test suite, e2e snapshots 1,142 passed / 0 failed, IQL lint, corpus regeneration byte-identical, per-scenario ledger, secret scan.
jsam
force-pushed
the
verified-completions-plan
branch
from
August 14, 2026 06:57
d346345 to
f3af72b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The foundation of Verified Completions (tracking issue #89), shipped as one verified unit: design, engine-validated rule pack, executable benchmark with per-scenario verification, tests at every level, and the docs that explain it.
The proven core loop
Conversation text becomes atomic claims with quoted evidence, claims become facts in a per-conversation knowledge graph, and the consistency rule pack derives findings with proof trees.
poc/poc_verify.pyruns that loop end to end (extract -> validate -> load -> judge); the scorer and ingestion validator are negative-tested (fabricated quotes get dropped, impossible expectations fail rows).The rule pack, validated on the real engine (#81)
eqidentity helper symmetrized at the base so.whyproof queries terminateinverse_functional;attr_domainseeded so the domain rule can fireexamples/iql/42_consistency_pack/), full suite greenThe benchmark (corpus v2.1)
poc/corpus.pydeterministically generates 1,628 scenarios across 16 families (~100 each, all 48 sub-variants at n>=33) covering value, negation, temporal, spatial, causal, structural, numeric, counting, identity, classification, and instruction corruption, plus 102 controls; 449 scenarios plant the conflict 12+ turns deep. Every sample carries multi-valued category labels, exact conflicting spans, a clean twin, a natural task, and extractor-truth facts. The corpus is documented for consumers inpoc/DATASET.md(design principles, scenario anatomy, per-scenario validation, how to run and extend it) and regenerates byte for byte.Per-scenario verification (
poc/verify_each.py, ledger committed): spans/clean-twins/labels 1,628/1,628; engine pass 1,526/1,526 with EXACT finding-kind match - the engine fires precisely the expected kinds and nothing else, on every corrupted scenario, deterministically.Completed behavioral study (corpus v1, 12 families at n=30, Wilson CIs, double-graded): corruption drops sound outputs from 95% to 87% [83-90] - 38 silent commits to arbitrary values, 4 contradictions written into deliverables; attaching the InputLayer finding restores 98% [96-99]. Corrupted system prompts are the standout: 17% sound alone -> 93% with the finding. Raw data archived in
poc/results/archive_v1_full_bench_390.json. The v2.1 LLM re-baseline continues in #95 (resume-until-verified tooling is in this PR; 100/1,628 done).Also in this unit
poc/full_bench.py): asked-to-check vs asked-a-real-task, double grading with a different-model arbiter (Opus) on disagreements, tiered runs (smoke/standard/full),--resume, subscription-OAuth-first authscripts/run_snapshot_tests.sh): test server starts from a generated config with WS rate limits off; client key read from the credentials file (--configmode rejects the old env var - Config: documented auth env vars break --config startup; default path ignores env overrides; stale config.toml template #92)docs/COVERAGE-AUDIT.md) mapping the full space of logic-resolvable inconsistency; gaps tracked in Benchmark coverage: close the gaps found by the consistency-type audit #94docs/internals/verified-completions/README.md, mermaid diagrams) and researcher-grade ontology spec (docs/ontology.md: extraction contract, type system, every verification rule verbatim)docs/blog/consistency-ontology.md): the ontology as a build-it-yourself journeyVerified on this exact tree
cargo fmt, clippy (-D warnings), full cargo test suite, e2e snapshots (1,142 passed / 0 failed), IQL lint on pack + fixtures, corpus regeneration byte-identical, per-scenario ledger, secret scan clean.
Engine bugs found and filed along the way: #91 (silent empty multi-clause unions), #92 (config/env-var conflicts), #93 (inserts ack while storing nothing after data-dir loss).
Closes #81.