Skip to content

Verified Completions M4: hardening - entailment guard, pluggable extractor, tenant limits #87

Description

@jsam

Why

Two residual risks after M3, plus operational hardening for untrusted traffic.

Risk one: the extractor invents a claim the source text does not support. Every claim already carries a verbatim quote, so a human can check any finding in seconds, but nothing machine-checks that the quote actually entails the claim.

Risk two: dependence on the Anthropic API for extraction. The extractor should sit behind a trait so a local model can slot in later without touching the pipeline.

What to do

  • Entailment guard: a small NLI (natural language inference) model checks each claim against its quoted span at ingestion; claims the quote does not support are dropped and counted, same as other validator rejections
  • Extractor trait: the Anthropic-backed extractor becomes one implementation; add a config hook for a local extraction model behind the same interface
  • Additional upstream completion providers if there is demand (the OpenAI-compatible surface makes this mostly parameter mapping)
  • Multi-tenant limits: per-key quotas and per-KG memory ceilings. max_sessions alone is too blunt once untrusted traffic is in front of this
  • Session purge route DELETE /v1/sessions/{id} and a documented data-retention story (message contents not logged by default)

Done when

Scoped per item; this is a hardening bucket, not a single deliverable. Split into separate issues when M3 lands if that works better.

Depends on

  • Output verification (M3)

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendBackend/server relatedverified-completionsVerified Completions: OpenAI-compatible endpoint with consistency checking

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions