Architecture

CSharpDB is built in clean layers, each with a single responsibility. From client access down to file I/O.

Need the full source guide? The original long-form markdown version is preserved as Architecture Source Reference.

Layered Architecture

Client & Access Layer

ADO.NET, Client SDK (HTTP/gRPC/Direct), REST API, CLI, MCP Server, VS Code Extension

▼

Application Layer

Admin UI (Blazor), Database class, Collection<T> API

▼

Query Execution

SQL Tokenizer, Parser, Query Planner, Operators, Expression Evaluator

▼

Pipeline & Procedures

PipelineOrchestrator, Sources, Transforms, Destinations, Stored Procedures

▼

Catalog & Indexing

SchemaCatalog, BTreeIndexStore, CachingIndexStore, collection path indexes

▼

Storage Engine

BTree, Pager, RecordEncoder, TransactionCoordinator

▼

Durability & Recovery

WriteAheadLog, WalIndex, CheckpointCoordinator, Snapshot Readers

▼

Device / File I/O

FileStream, LRU PageCache, DictionaryPageCache (in-memory mode)

Embedded-to-gRPC Runtime Boundary

CSharpDB has one database engine and one public client contract across local and remote shapes. Embedded mode keeps the engine in the application process. gRPC mode moves file ownership into CSharpDB.Daemon and lets applications call the same ICSharpDbClient contract through Transport = Grpc.

flowchart TB
    App["Application Code"]

    subgraph Embedded["Embedded Mode"]
        DirectClient["CSharpDbClient<br/>Transport = Direct"]
        Engine["CSharpDB.Engine<br/>Database"]
        Storage["Storage Stack<br/>SchemaCatalog, B+Tree, Pager, WAL"]
        File[".db + .wal files"]
    end

    subgraph Remote["gRPC Mode"]
        GrpcClient["CSharpDbClient<br/>Transport = Grpc"]
        Daemon["CSharpDB.Daemon<br/>ASP.NET Core + gRPC"]
        RpcService["CSharpDbRpcService"]
        HostClient["ICSharpDbClient<br/>Direct inside daemon"]
    end

    App --> DirectClient --> Engine --> Storage --> File
    App --> GrpcClient --> Daemon --> RpcService --> HostClient --> Engine

Boundary rule: gRPC is transport and process isolation, not a second database engine. The daemon opens one configured database file through a direct ICSharpDbClient, keeps that runtime warm, and exposes generated RPC methods over the same client-facing operation set.

sequenceDiagram
    participant App
    participant Client as GrpcTransportClient
    participant RPC as CSharpDbRpcService
    participant Host as Daemon ICSharpDbClient
    participant DB as Database Engine
    participant WAL as Pager/WAL/File

    App->>Client: ExecuteSqlAsync(sql)
    Client->>RPC: ExecuteSql(SqlRequest)
    RPC->>Host: ExecuteSqlAsync(sql)
    Host->>DB: Parse/plan/execute
    DB->>WAL: Read/write pages, commit via WAL
    WAL-->>DB: Durable result
    DB-->>Host: SqlExecutionResult
    Host-->>RPC: Model result
    RPC-->>Client: SqlExecutionResultMessage
    Client-->>App: SqlExecutionResult

Scenarios

Scenario	Use	Recommended entry point
Single embedded app	One process safely owns the database file.	`Database.OpenAsync(...)` or direct `CSharpDbClient`
Embedded with client abstraction	App code should stay transport-neutral.	`CSharpDbClientOptions { DataSource = "app.db" }`
Local daemon / sidecar	Multiple local processes or tools need one warm owner.	`Transport = CSharpDbTransport.Grpc`, localhost daemon endpoint
Internal service host	Trusted backend services share one database owner.	gRPC daemon with API key, TLS termination, and long-lived clients
Admin and tooling access	UI, CLI, VS Code, REST, and gRPC clients share the same runtime.	Point tools at the daemon instead of opening the file directly
Not a fit	Public internet, multi-tenant, multi-database, or distributed write coordination.	Build an outer service layer before exposing it broadly

Cross-Cutting Observability Plane

Observability follows the same runtime ownership boundary as database work. Engine, client, data-provider, and storage paths emit through the BCL-only CSharpDB.Observability vocabulary. Embedded applications may attach listeners directly; API and daemon hosts adapt the same sources into ILogger, OpenTelemetry/OTLP, Prometheus, health routes, and remote diagnostics. Exporter packages and ASP.NET Core routes do not flow back into the engine dependency graph.

flowchart LR
    Runtime["Engine / Client / Data / Storage"]
    Contract["CSharpDB.Observability<br/>DiagnosticListener + ActivitySource + Meter<br/>versioned runtime snapshots"]
    Host["Application, API, or Daemon host"]
    Logging["ILogger"]
    OTel["OpenTelemetry / OTLP"]
    Prom["Prometheus"]
    Remote["HTTP / gRPC diagnostics"]
    Admin["Admin Observability workspace"]

    Runtime --> Contract --> Host
    Host --> Logging
    Host --> OTel
    Host --> Prom
    Host --> Remote --> Admin

Direct, HTTP, gRPC, and sharded clients preserve the same schema version, capture time, server instance id, counter epoch, scope, availability, and truncation metadata. A sharded coordinator retains per-shard identity instead of combining counters across unrelated process lifetimes. Ordinary telemetry omits SQL, values, credentials, connection strings, paths, and exception messages. See Observability and Diagnostics.

Page Format (4096 bytes)

Every page uses a slotted format with a fixed header, cell pointer array growing forward, and cell data growing backward from the end of the page.

Page Header
Type, Cell Count, Free Offset

Cell Pointer Array →

Free Space

← Cell Data

Header Pointers (grow →) Free space Cells (grow ←)

WAL File Format

The Write-Ahead Log consists of a 32-byte header followed by a sequence of frames. Each frame contains a 24-byte header and a full 4096-byte page image.

WAL Header (32 bytes)
Magic "CWAL", Version, Page Size, Salts, Checksum Seed

Frame Header (24 bytes)
Page Number, DB Page Count, Salt1, Salt2, Checksum

Page Image (4096 bytes)

Frame Header (24 bytes)

Page Image (4096 bytes)

...

Transaction Flow

Writer Path

Acquire exclusive lock

Modify pages in memory

Write dirty pages to WAL

Mark commit frame

Update WAL index

Release lock

Checkpoint (if policy triggers)

Reader Path

Capture WAL snapshot

Create read-only pager

Read pages (WAL first, then DB file)

Return consistent results

Release snapshot on dispose

Database Modes

Mode	Entry Point	Characteristics
Default	`Database.OpenAsync(...)`	Standard file-backed open with full durability.
In-Memory	`Database.OpenInMemoryAsync(...)`	Dictionary-based page cache. No file I/O. Fast but non-persistent.
Hybrid	`Database.OpenHybridAsync(...)`	Lazy-resident pages with durable backing file. On-demand loading.

Project Structure

Assembly	Purpose
`CSharpDB.Primitives`	Shared types: DbValue, DbType, Schema, ErrorCodes
`CSharpDB.Storage`	B+tree, Pager, WAL, page cache, serialization
`CSharpDB.Storage.Diagnostics`	Database/WAL/Index inspection
`CSharpDB.Observability`	BCL telemetry contracts, bounded runtime models, source-generated wire schema
`CSharpDB.Sql`	SQL tokenizer, parser, AST
`CSharpDB.Execution`	Query planner, operators, expression evaluator
`CSharpDB.Engine`	Database class, Collection<T>, top-level API
`CSharpDB.Pipelines`	ETL orchestrator, sources, transforms, destinations
`CSharpDB.Data`	ADO.NET provider
`CSharpDB.Client`	Unified client SDK with pluggable transports
`CSharpDB.Api`	ASP.NET Core REST API
`CSharpDB.Daemon`	gRPC service host
`CSharpDB.Cli`	Interactive REPL shell
`CSharpDB.Admin`	Blazor Server dashboard
`CSharpDB.Mcp`	Model Context Protocol server for AI
`CSharpDB.Generators`	Source generators for trim-safe typed collections
`CSharpDB.Native`	NativeAOT C library for FFI