Skip to content

Git-native KB: commit-time projection, direct-caller adapter, store facade - #1663

Merged
CREDO23 merged 26 commits into
MODSetter:devfrom
CREDO23:kb-commit-time-projection
Aug 5, 2026
Merged

Git-native KB: commit-time projection, direct-caller adapter, store facade#1663
CREDO23 merged 26 commits into
MODSetter:devfrom
CREDO23:kb-commit-time-projection

Conversation

@CREDO23

@CREDO23 CREDO23 commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Continues the git-native KB rollout (backend only, behind the per-workspace flag).

  • Phase 6 — project documents/folders rows at commit time so agent-authored notes appear in the UI immediately; chunking/embedding stays async.
  • Phase 7 — one direct-caller adapter for every non-agent writer (deletes, moves, creates), fixing the UI-delete-leaves-the-file resurrection the canary found.
  • Phase 8 (design + facade) — fold the store into a single KnowledgeStore facade, and lock the path law: id is identity, the path is an authored-once label, folders start in the store (.keep for empty ones).

Docs: plans/git-native-kb/06..08. No production workspace flipped.

High-level PR Summary

This PR advances the git-native knowledge base backend across three phases. Phase 6 splits row projection out of the indexer to run at commit time, so agent-authored notes appear in the UI immediately (before chunking/embedding). Phase 7 wires every non-agent writer (deletes, moves, creates) through a direct-caller adapter to fix a resurrection bug where UI deletes left files behind. Phase 8 establishes the path law (id is identity, path is authored-once), introduces a unified KnowledgeStore facade consolidating all write capabilities, and prepares the migration seed to canonically re-author every path (.xml.md) per workspace. The changes include major refactoring: document_revision_recorder.py deleted in favor of service.py, path logic consolidated into paths.py, and row primitives extracted into index/rows.py to be shared by both the projector and indexer.

⏱️ Estimated Review Time: 1-3 hours

💡 Review Order Suggestion
Order File Path
1 plans/git-native-kb/00-umbrella-plan.md
2 plans/git-native-kb/06-zero-projection.md
3 plans/git-native-kb/07-direct-caller-adapter.md
4 plans/git-native-kb/08-store-facade-and-paths.md
5 plans/git-native-kb/00c-shared-contract.md
6 plans/git-native-kb/05-migration.md
7 surfsense_backend/app/knowledge_store/paths.py
8 surfsense_backend/app/knowledge_store/schemas/__init__.py
9 surfsense_backend/app/knowledge_store/schemas/revision.py
10 surfsense_backend/app/knowledge_store/schemas/working_copy.py
11 surfsense_backend/app/knowledge_store/schemas/outcome.py
12 surfsense_backend/app/knowledge_store/exceptions.py
13 surfsense_backend/app/knowledge_store/factory.py
14 surfsense_backend/app/knowledge_store/engines/base.py
15 surfsense_backend/app/knowledge_store/__init__.py
16 surfsense_backend/app/knowledge_store/service.py
17 surfsense_backend/app/knowledge_store/index/rows.py
18 surfsense_backend/app/knowledge_store/index/project.py
19 surfsense_backend/app/knowledge_store/index/converge.py
20 surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/middleware/knowledge_store_persistence/commit_turn.py
21 surfsense_backend/app/agents/chat/multi_agent_chat/shared/middleware/filesystem/backends/git_tree.py
22 surfsense_backend/app/services/folder_service.py
23 surfsense_backend/app/routes/documents_routes.py
24 surfsense_backend/app/routes/notes_routes.py
25 surfsense_backend/app/routes/search_source_connectors_routes.py
26 surfsense_backend/app/tasks/celery_tasks/document_tasks.py
27 surfsense_backend/app/routes/editor_routes.py
28 surfsense_backend/app/indexing_pipeline/indexing_pipeline_service.py
29 surfsense_backend/app/observability/metrics.py
30 surfsense_backend/app/knowledge_store/locks.py
31 surfsense_backend/app/knowledge_store/janitor.py
32 surfsense_backend/app/knowledge_store/migrate.py
33 surfsense_backend/scripts/migrate_knowledge_store.py
34 surfsense_backend/app/tasks/celery_tasks/knowledge_store/index_tasks.py
35 surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/middleware/kb_persistence/middleware.py
36 surfsense_backend/tests/integration/knowledge_store/test_service.py
37 surfsense_backend/tests/integration/knowledge_store/index/test_converge.py
38 surfsense_backend/tests/integration/knowledge_store/index/test_project.py
39 surfsense_backend/tests/integration/knowledge_store/test_commit_turn.py
40 surfsense_backend/tests/integration/knowledge_store/test_store.py
41 surfsense_backend/tests/integration/knowledge_store/test_write_lock.py
42 surfsense_backend/tests/integration/knowledge_store/conftest.py
43 surfsense_backend/tests/unit/knowledge_store/index/test_index_tasks.py

Need help? Join our Discord

CREDO23 added 26 commits July 31, 2026 15:12
The derived index needs it too, and importing it from the agent middleware
closed an import cycle. kb_persistence is deleted at the git-native cut, so the
helper had to outlive it either way.
Row upsert, rename following, resolution and pruning move to index/rows.py
unchanged, so a second writer can share one notion of document identity.
Nothing in there imports the indexing pipeline: the commit path is about to pay
this module's import cost on every save.
The sidebar needs a document row, and its id, before it can show a note. Phase 4
built that row inside the indexer, which put it behind chunking, embedding and a
queue hop, so a note the agent had already committed took seconds to appear and
the optimistic overlay had no id to work with.

The commit path now writes the row from its own change list and dispatches
document_created/updated/deleted naming real ids; chunks and vectors stay async.
The projection takes the index lock, stands aside when a rebuild holds it, and
never stamps last_indexed_revision — the content is not indexed yet.
Phase 6 planned to fold the projection into the indexer and Phase 4 did; the
canary showed why that couples UI freshness to embedding latency. Marks the
phase shipped with the reversal, and notes the one gap left behind: an emptied
folders row is never pruned, because no diff announces a folder emptying.
The canary turned up four files in the store with no row behind them.
DELETE /documents/{id} hands off to a task that never learned about the
store, so the row goes and the file stays -- the deferred REST adapter's
missing half.

That inverts the drift monitor: index_tree treats the surviving file as
truth, re-creates the document, and then reports ok, having just brought
back content a user deleted. Flipping real workspaces on that is how
deletions come back.
The delete gap was not one endpoint. Six HTTP writers reach git; about
twenty do not -- every delete, every move, and the creates that skip the
ingestion pipeline.

Twenty handlers each remembering to call the recorder is how six got
wired and twenty did not, so the fix is a verb on the adapter and callers
that hand it documents rather than paths. Deletes and moves have to
record before the Postgres commit: the path is read from the row that is
about to stop existing.
The adapter could only record content arriving, so every caller that
deleted or moved a document had nowhere to say so and simply did not.

Deletes read the path off the row, which means they have to record before
the row goes; the verb owns that ordering so no caller has to know it.
Moves record as tx.move rather than a remove and a write, because the
indexer recovers a renamed document by detecting the rename -- an id that
churns takes saved citations and version history with it.

Callers hand over documents, never paths. A caller that recomputes the
path is a caller whose delete misses the file the agent named.
The store's consumer-facing verbs sat in app/services as a loose file
named after the mechanism, while the rest of the subsystem lived in
app/knowledge_store. Callers had to know both places, and the next verb
would have been guessed into whichever the author saw first.

Now it reads like file_storage: the package owns its application service,
and consumers import one module. Also says so in the docstring, since the
reason a caller reaches past a facade is usually that nobody told them it
was one.
Deleting a document dropped the row and left its file in the store, so
the next whole-tree rebuild read the file back and re-created the
document -- and the drift check then reported ok, having just resurrected
content someone deleted. Four such files turned up in the canary.

The single delete and the folder-subtree delete ran near-identical purge
loops, so the recording went into one extracted helper rather than into
both. It runs before the rows go, because the path is read off the row.
The note delete and the two desktop-sync sweeps deleted rows inline, so
they left files behind the same way the document delete did. Both sweeps
already knew their whole batch, so they now record it as one revision
instead of one per file.

The regression test goes through the indexer rather than the store: the
symptom users would report is not a stray file, it is a document they
deleted reappearing after a rebuild.
Both issue Core-level DELETE statements, so they see no ORM object and
were the reason a session-event hook could not have covered this phase
by itself.

The connector delete now loads each batch as rows -- only the columns
that locate a file, since the content columns are the large ones -- and
records before deleting. The workspace delete drops the store outright:
there is nothing left to record a revision into, and removing ten
thousand paths one at a time writes a history that dies with it.
Locks one naming/resolution law for both the git tree and the Postgres
rows: id is identity, the path is an authored-once label. Folders start
in the store and materialize empty ones with a .keep, so an empty folder
survives a rebuild instead of leaving an orphan row. The migration seed
is the per-workspace vehicle that heals the old .xml/id-suffix debt.
…d C1

Point the seed at the path law it applies, amend C1's .xml rule (the
unique_identifier_hash demotion lifts it), and note the folder law
closes the Phase-6 emptied-folder gap.
Move Change, Revision, TrackedPath, and WorkingCopy out of the engine
base into schemas/, and add the single Outcome every write capability
returns, so the facade and its consumers share one vocabulary.
… engine base

The engine now imports Revision/Change/TrackedPath/WorkingCopy rather
than defining them, so there is one definition to depend on.
One KnowledgeStoreError root with KnowledgeStoreLockError under it, so a
caller can catch the module's failures without importing lock internals.
The module now holds both the write and index locks and raises the
tagged KnowledgeStoreLockError; the name says what it is.
Absorb store_path.py and the surviving resolver parts, add the StorePath
value object and allocate_path (.md default, sanitize-once, (2) collisions,
never a doc id), and quarantine the legacy .xml derivation for the
kb_postgres renderers until the cut.
One place decides a workspace's storage engine, so the facade stays free
of the concrete GitContentEngine.
service.py is now the single entry point every consumer builds through
(for_workspace/as_user/as_agent/with_session), carrying the primitives,
the revise() transaction, the document capabilities, and the agent turn.
store.py is gone; scattered store access collapses into one surface.
commit_turn.py delegates the commit to KnowledgeStore.commit_turn and
keeps only its own work: the message, the receipts, and the announce,
all read off the returned Outcome.
…n_copy

The backend opens the turn's copy through the facade and imports
thread_working_copy_id from it, so the turn-copy convention lives with
the store that owns it.
converge/project/rows build the store through the facade and read its
Outcome, so the driven consumer never reaches store internals.
… the facade

The janitor, the seeder, the index tasks, and the fleet runner all reach
the store through service.py, closing out the old store/write_lock imports.
…on suite

Point the fixtures and the commit-turn/service/migrate/project tests at
service.py, and move the failure-injection monkeypatches onto the methods
that now do the work (KnowledgeStore._commit_files, app.db.shielded_async_session).
@vercel

vercel Bot commented Aug 5, 2026

Copy link
Copy Markdown

@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4c216e82-8958-4f40-8af3-98f5f5ffdf67

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@CREDO23
CREDO23 merged commit 2bfc220 into MODSetter:dev Aug 5, 2026
4 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant