Skip to content

Verified Completions M0: POST /v1/verify - extract facts, return consistency findings with proofs #83

Description

@jsam

Why

POST /v1/verify is the smallest useful piece of Verified Completions: send a conversation, get back a list of contradictions with quoted evidence and a proof tree. No completion is generated. This alone is enough to lint a system prompt in CI or batch-audit stored conversations, and it exercises the whole pipeline we need later for the full proxy.

How it works

  1. Client POSTs an OpenAI-style messages array
  2. The gateway calls Claude (Haiku, temperature 0, structured outputs so the response always matches our JSON schema) to turn the messages into small factual claims. Every claim must carry a verbatim quote from the message it came from
  3. A validator checks each claim before it touches the engine (see below)
  4. Valid claims become plain fact inserts into a per-conversation knowledge graph over the WebSocket API. The rule pack is loaded into that KG once at creation
  5. The gateway queries the findings, fetches .why proof trees, and returns the response block defined in the RFC (section 9)

What to do

  • Gateway crate skeleton: axum server, config sections [llm] and [verify] (see RFC section 4.1), route only mounted when an Anthropic API key is configured
  • Extraction call: Anthropic Messages API with structured outputs pinned to docs/internals/verified-completions/extraction/claim-schema.json, prompt caching on the static instruction prefix, using docs/internals/verified-completions/extraction/fact-lifecycle-prompt.md (NOT the superseded extraction-prompt.md)
  • Ingestion validator. Drop a claim (and count it in metrics) when:
    • its quote is not a verbatim substring of the message it points to
    • its message index is outside the batch being extracted
    • its id collides with an existing claim id
    • a value that claims to be a date does not parse as ISO-8601
    • it tries to override a seeded ontology entry (extending is allowed, overriding is not)
    • it duplicates an existing claim from the same message
  • Numeric mirroring: dates become YYYYMMDD integers, datetimes become epoch seconds, quantities become plain integers, all inserted into claim_num / constraint_num. The engine cannot order-compare strings, so every "X before Y" or "X exceeds limit" check rides these integer columns
  • Fact compilation: validated claims to parameterized IQL inserts over the WS API. Fixed relation set only. Conversation text must never be able to inject rules or queries
  • Findings assembly: query finding_src, walk the .why proof trees (already available as JSON over the wire), collapse mirrored pairs like (a,b)/(b,a) into one finding, build the consistency response block from RFC section 4.2
  • Fail open: any extractor or engine error returns status: "unverified" with a reason instead of failing the request

Done when

Demo works end to end: paste a conversation that contradicts itself, get back findings where each one shows the two quoted sentences and the rule that connects them.

Depends on

  • Rule pack validated on the real engine
  • Workspace split

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendBackend/server relatedenhancementNew feature or requestverified-completionsVerified Completions: OpenAI-compatible endpoint with consistency checking

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions