Git-native KB: commit-time projection, direct-caller adapter, store facade - #1663
Conversation
The derived index needs it too, and importing it from the agent middleware closed an import cycle. kb_persistence is deleted at the git-native cut, so the helper had to outlive it either way.
Row upsert, rename following, resolution and pruning move to index/rows.py unchanged, so a second writer can share one notion of document identity. Nothing in there imports the indexing pipeline: the commit path is about to pay this module's import cost on every save.
The sidebar needs a document row, and its id, before it can show a note. Phase 4 built that row inside the indexer, which put it behind chunking, embedding and a queue hop, so a note the agent had already committed took seconds to appear and the optimistic overlay had no id to work with. The commit path now writes the row from its own change list and dispatches document_created/updated/deleted naming real ids; chunks and vectors stay async. The projection takes the index lock, stands aside when a rebuild holds it, and never stamps last_indexed_revision — the content is not indexed yet.
Phase 6 planned to fold the projection into the indexer and Phase 4 did; the canary showed why that couples UI freshness to embedding latency. Marks the phase shipped with the reversal, and notes the one gap left behind: an emptied folders row is never pruned, because no diff announces a folder emptying.
The canary turned up four files in the store with no row behind them.
DELETE /documents/{id} hands off to a task that never learned about the
store, so the row goes and the file stays -- the deferred REST adapter's
missing half.
That inverts the drift monitor: index_tree treats the surviving file as
truth, re-creates the document, and then reports ok, having just brought
back content a user deleted. Flipping real workspaces on that is how
deletions come back.
The delete gap was not one endpoint. Six HTTP writers reach git; about twenty do not -- every delete, every move, and the creates that skip the ingestion pipeline. Twenty handlers each remembering to call the recorder is how six got wired and twenty did not, so the fix is a verb on the adapter and callers that hand it documents rather than paths. Deletes and moves have to record before the Postgres commit: the path is read from the row that is about to stop existing.
The adapter could only record content arriving, so every caller that deleted or moved a document had nowhere to say so and simply did not. Deletes read the path off the row, which means they have to record before the row goes; the verb owns that ordering so no caller has to know it. Moves record as tx.move rather than a remove and a write, because the indexer recovers a renamed document by detecting the rename -- an id that churns takes saved citations and version history with it. Callers hand over documents, never paths. A caller that recomputes the path is a caller whose delete misses the file the agent named.
The store's consumer-facing verbs sat in app/services as a loose file named after the mechanism, while the rest of the subsystem lived in app/knowledge_store. Callers had to know both places, and the next verb would have been guessed into whichever the author saw first. Now it reads like file_storage: the package owns its application service, and consumers import one module. Also says so in the docstring, since the reason a caller reaches past a facade is usually that nobody told them it was one.
Deleting a document dropped the row and left its file in the store, so the next whole-tree rebuild read the file back and re-created the document -- and the drift check then reported ok, having just resurrected content someone deleted. Four such files turned up in the canary. The single delete and the folder-subtree delete ran near-identical purge loops, so the recording went into one extracted helper rather than into both. It runs before the rows go, because the path is read off the row.
The note delete and the two desktop-sync sweeps deleted rows inline, so they left files behind the same way the document delete did. Both sweeps already knew their whole batch, so they now record it as one revision instead of one per file. The regression test goes through the indexer rather than the store: the symptom users would report is not a stray file, it is a document they deleted reappearing after a rebuild.
Both issue Core-level DELETE statements, so they see no ORM object and were the reason a session-event hook could not have covered this phase by itself. The connector delete now loads each batch as rows -- only the columns that locate a file, since the content columns are the large ones -- and records before deleting. The workspace delete drops the store outright: there is nothing left to record a revision into, and removing ten thousand paths one at a time writes a history that dies with it.
Locks one naming/resolution law for both the git tree and the Postgres rows: id is identity, the path is an authored-once label. Folders start in the store and materialize empty ones with a .keep, so an empty folder survives a rebuild instead of leaving an orphan row. The migration seed is the per-workspace vehicle that heals the old .xml/id-suffix debt.
…d C1 Point the seed at the path law it applies, amend C1's .xml rule (the unique_identifier_hash demotion lifts it), and note the folder law closes the Phase-6 emptied-folder gap.
Move Change, Revision, TrackedPath, and WorkingCopy out of the engine base into schemas/, and add the single Outcome every write capability returns, so the facade and its consumers share one vocabulary.
… engine base The engine now imports Revision/Change/TrackedPath/WorkingCopy rather than defining them, so there is one definition to depend on.
One KnowledgeStoreError root with KnowledgeStoreLockError under it, so a caller can catch the module's failures without importing lock internals.
The module now holds both the write and index locks and raises the tagged KnowledgeStoreLockError; the name says what it is.
Absorb store_path.py and the surviving resolver parts, add the StorePath value object and allocate_path (.md default, sanitize-once, (2) collisions, never a doc id), and quarantine the legacy .xml derivation for the kb_postgres renderers until the cut.
One place decides a workspace's storage engine, so the facade stays free of the concrete GitContentEngine.
service.py is now the single entry point every consumer builds through (for_workspace/as_user/as_agent/with_session), carrying the primitives, the revise() transaction, the document capabilities, and the agent turn. store.py is gone; scattered store access collapses into one surface.
commit_turn.py delegates the commit to KnowledgeStore.commit_turn and keeps only its own work: the message, the receipts, and the announce, all read off the returned Outcome.
…n_copy The backend opens the turn's copy through the facade and imports thread_working_copy_id from it, so the turn-copy convention lives with the store that owns it.
converge/project/rows build the store through the facade and read its Outcome, so the driven consumer never reaches store internals.
… the facade The janitor, the seeder, the index tasks, and the fleet runner all reach the store through service.py, closing out the old store/write_lock imports.
…on suite Point the fixtures and the commit-turn/service/migrate/project tests at service.py, and move the failure-injection monkeypatches onto the methods that now do the work (KnowledgeStore._commit_files, app.db.shielded_async_session).
|
@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Continues the git-native KB rollout (backend only, behind the per-workspace flag).
documents/foldersrows at commit time so agent-authored notes appear in the UI immediately; chunking/embedding stays async.KnowledgeStorefacade, and lock the path law: id is identity, the path is an authored-once label, folders start in the store (.keepfor empty ones).Docs:
plans/git-native-kb/06..08. No production workspace flipped.High-level PR Summary
This PR advances the git-native knowledge base backend across three phases. Phase 6 splits row projection out of the indexer to run at commit time, so agent-authored notes appear in the UI immediately (before chunking/embedding). Phase 7 wires every non-agent writer (deletes, moves, creates) through a direct-caller adapter to fix a resurrection bug where UI deletes left files behind. Phase 8 establishes the path law (id is identity, path is authored-once), introduces a unified
KnowledgeStorefacade consolidating all write capabilities, and prepares the migration seed to canonically re-author every path (.xml→.md) per workspace. The changes include major refactoring:document_revision_recorder.pydeleted in favor ofservice.py, path logic consolidated intopaths.py, and row primitives extracted intoindex/rows.pyto be shared by both the projector and indexer.⏱️ Estimated Review Time: 1-3 hours
💡 Review Order Suggestion
plans/git-native-kb/00-umbrella-plan.mdplans/git-native-kb/06-zero-projection.mdplans/git-native-kb/07-direct-caller-adapter.mdplans/git-native-kb/08-store-facade-and-paths.mdplans/git-native-kb/00c-shared-contract.mdplans/git-native-kb/05-migration.mdsurfsense_backend/app/knowledge_store/paths.pysurfsense_backend/app/knowledge_store/schemas/__init__.pysurfsense_backend/app/knowledge_store/schemas/revision.pysurfsense_backend/app/knowledge_store/schemas/working_copy.pysurfsense_backend/app/knowledge_store/schemas/outcome.pysurfsense_backend/app/knowledge_store/exceptions.pysurfsense_backend/app/knowledge_store/factory.pysurfsense_backend/app/knowledge_store/engines/base.pysurfsense_backend/app/knowledge_store/__init__.pysurfsense_backend/app/knowledge_store/service.pysurfsense_backend/app/knowledge_store/index/rows.pysurfsense_backend/app/knowledge_store/index/project.pysurfsense_backend/app/knowledge_store/index/converge.pysurfsense_backend/app/agents/chat/multi_agent_chat/main_agent/middleware/knowledge_store_persistence/commit_turn.pysurfsense_backend/app/agents/chat/multi_agent_chat/shared/middleware/filesystem/backends/git_tree.pysurfsense_backend/app/services/folder_service.pysurfsense_backend/app/routes/documents_routes.pysurfsense_backend/app/routes/notes_routes.pysurfsense_backend/app/routes/search_source_connectors_routes.pysurfsense_backend/app/tasks/celery_tasks/document_tasks.pysurfsense_backend/app/routes/editor_routes.pysurfsense_backend/app/indexing_pipeline/indexing_pipeline_service.pysurfsense_backend/app/observability/metrics.pysurfsense_backend/app/knowledge_store/locks.pysurfsense_backend/app/knowledge_store/janitor.pysurfsense_backend/app/knowledge_store/migrate.pysurfsense_backend/scripts/migrate_knowledge_store.pysurfsense_backend/app/tasks/celery_tasks/knowledge_store/index_tasks.pysurfsense_backend/app/agents/chat/multi_agent_chat/main_agent/middleware/kb_persistence/middleware.pysurfsense_backend/tests/integration/knowledge_store/test_service.pysurfsense_backend/tests/integration/knowledge_store/index/test_converge.pysurfsense_backend/tests/integration/knowledge_store/index/test_project.pysurfsense_backend/tests/integration/knowledge_store/test_commit_turn.pysurfsense_backend/tests/integration/knowledge_store/test_store.pysurfsense_backend/tests/integration/knowledge_store/test_write_lock.pysurfsense_backend/tests/integration/knowledge_store/conftest.pysurfsense_backend/tests/unit/knowledge_store/index/test_index_tasks.py