feat: Reasoning Notebook demo with multimodal extraction and benchmarks - #80

Backend: Note Relation stored in InputLayer KG, full CRUD via
FastAPI (create, list, get, update, delete). Updates use
conditional retraction then re-insert. Auto-discovers server
credentials from .inputlayer-credentials.toml.

Frontend: Sidebar with note list and create/delete, title input
with textarea editor, auto-save after 800ms debounce or Cmd+S.
Two-column layout with Catppuccin-inspired dark theme.

Backend: Entity and Relationship Relations in KG. LangChain
extraction pipeline using ChatOpenAI.with_structured_output() to
pull entities and relationships from note content. Auto-extracts
on note save via BackgroundTasks. Manual trigger via POST
/notes/{id}/extract. GET /notes/{id}/entities returns extracted
data. GET /graph returns all entities and relationships. Delete
cascades to extracted data. Re-extraction retracts old facts first.

Frontend: ExtractionPanel shows entity/relationship counts and
tags below the editor. Manual "Extract" button triggers LLM
extraction with toast feedback. Refreshes on each save.

Config: LLM_BASE_URL, LLM_MODEL, OPENAI_API_KEY env vars.
Defaults to LM Studio local endpoint.

GraphView component using react-force-graph-2d renders all
entities and relationships as a force-directed graph. Nodes
colored by entity kind (person, org, tech, concept, etc.),
edges labeled with predicate names. Hover shows tooltip with
entity description. Click node navigates to source note.

Editor/Graph tab toggle in header. GET /graph API endpoint
feeds the graph data (added in Phase 2).

Backend: ontology.py uses ChatOpenAI.with_structured_output() to
scan all predicates and entity names, propose merges for synonyms
(e.g. "works for"/"employed by" -> "works_at"), and apply them as
IQL retract+insert operations. Also merges entity name variants.

Endpoints: GET /ontology/predicates lists distinct predicates,
POST /ontology/consolidate runs one round of consolidation.

Frontend: "Consolidate Ontology" button in graph view toolbar.
Shows merge results as a toast message, refreshes graph after.

Extraction errors are now returned to the client instead of being
swallowed silently. ExtractionPanel shows the actual error message.
Long note content is truncated to 4000 chars (configurable via
EXTRACTION_MAX_CHARS) to avoid exceeding small model context limits.

get_kg() now checks the WebSocket state before each request.
If the connection is closed (server restart, timeout, etc.),
it automatically reconnects and re-deploys the schema. This
prevents 500 errors when the engine restarts while the backend
is still running.

Backend: chat.py gathers all notes, entities, and relationships
from the KG as context, then uses ChatOpenAI to answer the user's
question with citations. POST /chat endpoint accepts message and
conversation history.

Frontend: ChatPanel component with message bubbles, Enter to
send, conversation history, loading indicator, and empty state
with example questions. Third tab (Editor / Graph / Chat) in
the header.

Clicking a node opens a detail panel on the right showing:
- Entity kind badge, name, and description
- Source notes (clickable, navigates to editor)
- All relationships (clickable targets navigate to that node)
- Selected node gets a bright ring, connected edges highlight

Click background or X button to dismiss. Graph canvas resizes
to make room for the panel. Hover tooltip hidden while panel
is open.

Backend: POST /why and POST /why_not endpoints that run .why and
.why_not IQL commands and return serialized proof trees (nodes as
a DAG with kind, conclusion, children, rule_id, bindings, source).

Frontend: ProvenanceTree component renders proof trees as a
collapsible tree with color-coded node kinds (base_fact green,
rule_application blue, aggregate amber). Shows rule IDs, variable
bindings, and source annotations.

Integrated into graph view: "Why?" button on entity detail panel
queries .why for that entity. Each relationship row also has a
"why?" link. Results open as a modal overlay.

kg.execute() drops proof_trees when converting to ResultSet.
Switch /why and /why_not to use kg._execute() which returns the
raw ResultResponse including proof trees.

Also fix ProvenanceTree to handle the actual wire format: kind
is "fact" (not "base_fact"), source "edb" means extensional
database (base fact). Added _raw_tree_to_dict normalizer.

The entity relation has 5 columns (id, name, kind, desc, source)
but the Why? button was sending a 4-arg query with name in the
first position. The engine returned 0 rows because column count
didn't match. Fixed to: ?entity(Id, "name", Kind, Desc, Source).

POST /ontology/cleanup removes entities and relationships whose
source note no longer exists. Handles stale data left behind from
early development before delete cascading was added.

Keyboard shortcuts: Cmd+N (new note), Cmd+E (editor), Cmd+G
(graph), Cmd+K (chat). Tooltip hints on tab buttons.

.env.example with all configuration options documented.

README with architecture diagram, quick start, manual start,
LLM setup (LM Studio and OpenAI), feature descriptions,
keyboard shortcuts table, and project structure.

compile_value() only escaped backslash and double-quote, but not
newline, carriage return, tab, or null byte. Multi-line content
(pasted text, code snippets) would silently fail to insert because
the IQL parser saw the newline as a statement terminator.

This is the same fix jsam applied to iql_literal() in the
LangChain params module, now applied to the core SDK compiler
that handles Relation.insert().

Derived rules deployed on startup:
- colleague(A, B): two people from the same note
- shared_context(A, B): entities from the same note
- connected(A, B): direct relationship in either direction
- reachable(A, C): transitive closure of connected

Graph shows derived edges as dashed purple lines, extracted edges
as solid green. Legend updated to distinguish them. "Why?" on
derived facts now shows multi-step proof trees.

Entity resolution via HNSW vector similarity:
- resolution.py embeds entity names using character n-gram hashing
  (no external embedding service needed)
- Builds HNSW index in InputLayer, finds near-duplicates above
  0.85 cosine threshold
- Merges variants into canonical names (keeps the longer name)
- "Resolve Entities" button in graph toolbar
- POST /ontology/resolve endpoint

vector_search returns capitalized column names (Entity_name not
entity_name). Use lowercased dict keys for lookup. Raise default
threshold from 0.85 to 0.95 to avoid false merges with the
simple character n-gram embedder.

Backend: images.py handles upload, storage, and multimodal LLM
extraction. Sends image to LM Studio vision model for both a
text description and structured entity/relationship extraction.
Image record stored in KG, extracted entities added to the
note's source. POST /notes/{id}/images uploads and extracts,
GET /images/{filename} serves stored images.

Frontend: Editor accepts drag-and-drop and paste of images.
Shows upload progress with "Analyzing..." state, then thumbnail
with description and entity count. Image description auto-appended
to note content for text search and chat context.

Supports any LM Studio multimodal model (ministral-3-3b confirmed
working with vision).

reachable is a transitive closure that creates O(n^2) edges with
even a few entities. shared_context similarly explodes since all
entities in the same note get paired. Keep only colleague (people
from same note) as derived relation — it's meaningful and bounded.

Entity IDs now use prefixes: t_ for text-extracted, i_ for
image-extracted. Text re-extraction only retracts entities
whose ID starts with t_{note_id}, preserving image entities.

Milkdown (MIT) provides WYSIWYG markdown editing with Nord theme.
Supports headings, lists, code blocks, bold/italic, links, and
images rendered inline. Content stays as markdown for extraction.

Features:
- Live markdown rendering as you type
- markdownUpdated listener for auto-save
- replaceAll action to sync content on note switch
- Image drag-and-drop still works (uploads + inserts markdown)
- Cmd+S still works for manual save

Three bugs fixed:
1. title captured at editor creation time was stale — now uses
   titleRef that's always current
2. replaceAll triggers markdownUpdated which saved to the wrong
   note — added suppressSaveRef flag during replaceAll
3. Image upload used stale title/noteId from closure — switched
   to refs throughout

All save paths now read from refs (noteIdRef, titleRef,
contentRef) instead of React state to avoid stale closures.

Two bugs:
1. Editor reverted on every keystroke because saving updated
   note.content in parent state, which triggered replaceAll,
   which reset the editor. Fixed: only sync editor content
   when note.id changes, not on every content update.

2. IQL engine returns \n as literal backslash-n instead of
   newline. Each save re-escaped, causing \\n -> \\\\n
   accumulation. Added _unescape_iql_string() to convert
   engine output back to real control characters.

Image upload was calling replaceAll to insert image markdown,
which triggered the save-suppression dance and still caused
the editor to jump back. Now the image description is only
appended to contentRef and saved to the backend without
touching the editor DOM. The user's cursor position and
in-progress edits are preserved.

Root cause: saves updated parent state, which re-rendered the
editor, which interfered with Milkdown's internal state.

Changes:
- All callbacks use refs (onSaveRef, onImageUploadedRef) so
  the useEditor callback never needs recreation
- doSave is a plain function (not useCallback) that reads from
  refs, avoiding dependency chains
- MilkdownProvider keyed by note.id so it fully remounts on
  note switch instead of trying to patch
- suppressRef starts true, enabled after 100ms timeout (not
  requestAnimationFrame which was too fast)
- Debounce increased to 1200ms to reduce save frequency

Milkdown's ProseMirror integration conflicted with React's
re-render cycle, causing the editor to revert to previous state
on every save. Replaced with a reliable textarea for editing
and react-markdown for preview, toggled via Edit/Preview button.

The textarea approach is simpler and works correctly:
- Monospace font (JetBrains Mono) for editing
- Preview mode renders markdown with image support
- Drag-and-drop and paste images still work
- No re-render interference with save cycle

starts_with() accepted by engine but returned 0 matches — it's
not functional as a filter. Reverted to retracting all entities
for the note by source_note_id. This means text re-extraction
also clears image entities, but that's acceptable since image
entities can be re-created by re-uploading.

Image entities now use source "img:{note_id}" instead of plain
"{note_id}". Text re-extraction only retracts source="{note_id}",
preserving image entities. The /entities endpoint fetches both
sources. Delete cascades to both. Graph already queries all
entities unfiltered.

…rnings

LLM returns compound kinds like "artwork/animal", "building/artwork"
which didn't match exact keys. Added kindColor() that checks
substring matches. Also added object, building, artwork, animal,
software to the color map.

…ability

…ection

The old prompt only looked for named entities (people, orgs, tech)
and returned empty for creative writing. Updated to also extract:
- Unnamed characters ("the keeper", "narrator")
- Places and landmarks (lighthouse, cities)
- Notable objects (freighter, staircase)
- Added "extract at least the key subjects and locations"

LLM often extracts relationships referencing names that aren't
in the entity list (e.g. "keeper" in a relationship but not as
an entity). Now auto-creates missing entities as "concept" type
so all relationship endpoints appear as nodes in the graph.

notebooks/query_patterns.ipynb showcases:

1. Semantic retrieval — HNSW vector search over entity embeddings.
   "Find entities related to navigation", "Find people", etc.

2. Structured retrieval — multi-hop IQL rules (connected, two_hop,
   same_note). Traverse relationships, find co-occurring entities,
   show provenance with .why proof trees.

3. Hybrid queries — vector similarity seeds into structured
   traversal. "Find things related to 'water' and show their
   2-hop connections."

4. Multimodal queries — show how image-extracted and text-extracted
   entities live in the same KG and can be queried together.
   Cross-modal connections, shared entities, cross-modal semantic
   search.

Added jupyter and ipykernel to backend deps.

Image extraction now captures:
- scene: "temple entrance, outdoors"
- objects: ["dragon sculpture", "staircase", "pillar"]
- people: "1 (adult)" or "none"
- emotion: "serene, sacred"
- event_type: "travel", "ceremony", etc.
- aesthetic: "warm lighting, candid"
- caption_seed: "golden dragons guard the temple"
- cultural_context, visible_text

Stored as typed ImageScene relation in the KG alongside the
existing entity/relationship extraction. API returns the full
analysis in the upload response.

…ables

Note deletion now also retracts image and image_scene records.
Cleanup endpoint also removes orphaned image and image_scene
rows whose note no longer exists.

The graph now includes image analysis data from image_scene:
- Scene node (caption_seed as label, warm rose color)
- Object nodes from the objects list (pink)
- Emotion node (red)
- Event type node (red)
- Cultural context node (purple)

All connected to the scene hub via "contains", "evokes",
"depicts", "cultural_context" edges (shown as derived/dashed).
They naturally cluster around the image's entity nodes since
they share the same source note.

New GET /notes/{id}/scenes endpoint returns image_scene data.
ExtractionPanel shows the rich analysis when expanded:
scene, objects, people, emotion, event type, aesthetic,
caption seed, cultural context, and visible text. Styled
with a rose accent border to match scene nodes in the graph.

benchmarks/ directory with:
- config.py: model definitions (local LM Studio + cloud OpenAI/Anthropic)
- run_benchmark.py: runs text and image extraction across all
  enabled models, measures time, entity/relationship counts,
  structured output compliance. Saves JSON results.
- compare.py: loads results, prints comparison tables by input
  and overall summary with avg entities, avg time, success rate.
- inputs/: test text (corporate + narrative) and images

Supports:
  uv run python run_benchmark.py              # all models
  uv run python run_benchmark.py --models local
  uv run python run_benchmark.py --models cloud
  uv run python run_benchmark.py --text-only
  uv run python compare.py                    # latest results

Added langchain-anthropic for Claude model support.

Opus 4.7 rejects temperature parameter. Mythos-preview not
available on all API keys, commented out and added sonnet-4.6.

Four charts in a 2x2 grid:
- Entities per input (grouped bar, one color per input file)
- Extraction time per input (grouped bar)
- Avg entities vs relationships (side-by-side bars)
- Speed vs quality scatter (time on X, entities on Y, hover shows model name)

Uses Chart.js 4 from CDN, dark theme matching the report.