FRED - Flexible Resource Execution Daemon

A Go daemon for Manifest Network providers that manages the complete lease lifecycle with pluggable backend integration, event-driven provisioning, and automatic resource management.

• Lease Lifecycle Management: Watches chain events and orchestrates provisioning through backends
• Multi-Backend Support: Route leases to different backends based on exact SKU UUID list, distributing new provisions to the least-loaded matching backend (lowest allocated-CPU ratio)
• Event-Driven Architecture: Uses Watermill for internal event routing with retries and middleware
• Tenant Authentication API: HTTP/HTTPS API with ADR-036 signature verification for tenant access
• Periodic Withdrawals: Configurable scheduled withdrawal of accumulated fees from active leases
• Credit Monitoring: Tracks tenant credit balances and auto-closes leases when credit is depleted
• Cross-Provider Credit Detection: Responds to credit depletion events from other providers
• Live Operations: Restart containers or deploy new manifests (update) on active leases with full release history tracking
• Data Retention & Restore: Soft-delete a lease's volumes on close and restore them into a new lease within a grace window, optionally onto a different SKU tier
• Security: Rate limiting, request size limits, input validation, and optional TLS

                              MANIFEST CHAIN
                                    |
                                    | WebSocket (events)
                                    v
+------------------------------------------------------------------+
|                              FRED                                 |
|                                                                   |
|  +------------------+                                             |
|  | Event Subscriber |  (fan-out: each consumer gets all events)  |
|  | (WebSocket)      |-----+------------------+                    |
|  +------------------+     |                  |                    |
|                           v                  v                    |
|  +------------------+  +------------------+  +------------------+ |
|  | Event Bridge     |  | Watcher          |  | (other future    | |
|  | -> Watermill     |  | (cross-provider) |  |  consumers)      | |
|  +------------------+  +------------------+  +------------------+ |
|           |                                                       |
|           v                                                       |
|  +------------------+     +------------------+                    |
|  | Watermill Router |---->| Provision        |                    |
|  | (event routing)  |     | Manager          |                    |
|  +------------------+     +------------------+                    |
|                                   |                               |
|  +------------------+             |                               |
|  | API Server       |<------------+                               |
|  | (tenant access)  |             |                               |
|  +------------------+             v                               |
|                           +------------------+                    |
|                           | Backend Router   |                    |
|                           | (SKU routing +   |                    |
|                           |  least-loaded)   |                    |
|                           +------------------+                    |
|                                   |                               |
+------------------------------------------------------------------+
                                    |
              +---------------------+---------------------+
              v                     v                     v
      +---------------+     +---------------+     +---------------+
      |   Docker-1    |     |   Docker-2    |     |   Docker-3    |
      |   Backend     |     |   Backend     |     |   Backend     |
      | (skus: [uuid])|     | (skus: [uuid])|     | (skus: [uuid])|
      +---------------+     +---------------+     +---------------+

The Event Subscriber uses a fan-out pattern where each consumer (Event Bridge, Watcher, etc.) gets its own channel and receives all events independently. This ensures that:

• The provisioner never misses lease events
• The watcher always sees cross-provider credit depletion events
• New consumers can be added without affecting existing ones

sequenceDiagram
    participant T as Tenant
    participant C as Chain
    participant F as Fred
    participant B as Backend

    Note over T,B: Lease Creation & Provisioning
    T->>C: Create Lease (with SKU)
    C-->>F: lease_created event
    F->>F: Route by SKU to backend
    F->>B: POST /provision
    B->>B: Provision resource (async)
    B->>F: POST /callbacks/provision (success)
    F->>C: MsgAcknowledgeLease
    C-->>F: lease_acknowledged event

    Note over T,B: Tenant Access
    T->>T: Sign auth token (ADR-036)
    T->>F: GET /v1/leases/{uuid}/connection
    F->>F: Verify signature & lease ownership
    F->>B: GET /info/{uuid}
    F-->>T: Connection details

    Note over T,B: Restart (same manifest)
    T->>F: POST /v1/leases/{uuid}/restart
    F->>B: POST /restart
    B->>B: Stop, recreate containers (async)
    B->>F: POST /callbacks/provision (success)

    Note over T,B: Update (new manifest)
    T->>F: POST /v1/leases/{uuid}/update
    F->>B: POST /update
    B->>B: Pull image, replace containers (async)
    B->>F: POST /callbacks/provision (success)

    Note over T,B: Release History
    T->>F: GET /v1/leases/{uuid}/releases
    F->>B: GET /releases/{uuid}
    F-->>T: Release history

    Note over T,B: Lease Closure
    T->>C: Close Lease (or credit depleted)
    C-->>F: lease_closed event
    F->>B: POST /deprovision
    B->>B: Cleanup resources

# Build all binaries (providerd, mock-backend, docker-backend, k3s-backend)
make all

# Build only providerd
go build -o build/providerd ./cmd/providerd

# Build only mock-backend
go build -o build/mock-backend ./cmd/mock-backend

# Build only docker-backend
go build -o build/docker-backend ./cmd/docker-backend

# Build only k3s-backend
go build -o build/k3s-backend ./cmd/k3s-backend

k3s-backend is currently an experimental, non-functional scaffold (ENG-133): the binary boots, serves the backend HTTP contract, and signs/verifies callbacks, but its provisioner returns status=failed, error="not implemented" for every provision. It is not usable in production; real Kubernetes provisioning lands in ENG-134+.

For a one-shot dev environment against a running local chain, use:

bash scripts/dev-init.sh

This registers a provider and SKUs on-chain, generates config.docker.yaml (the providerd config) and docker-backend.yaml, and writes a callback secret. All settings are overridable via environment variables — see the script header for the full list. Requires manifestd, jq, curl, and openssl on PATH plus a running local chain.

Copy the example configuration and customize:

cp config.example.yaml config.yaml

All required fields are validated at startup. The daemon will fail to start with a clear error message if any required configuration is missing or invalid.

Backends are services that handle the actual resource provisioning. Each backend URL must be an absolute URL with http:// or https:// scheme.

Leases are routed to backends using the skus field — an exact list of on-chain SKU UUIDs. A backend with no skus matches nothing (use default: true for fallback). When multiple backends match the same SKU, Fred routes each new provision to the least-loaded matching backend — the SKU-matching backend reporting the lowest allocated-CPU ratio from its /stats endpoint (ENG-318). Ties break by fewest in-flight provisions, then by a round-robin counter; round-robin is also the fallback when no matching backend exposes usable load stats.

backends:
  # Give every backend the same skus list so they all match,
  # then Fred routes each provision to the least-loaded one.
  - name: docker-1
    url: "http://10.0.0.1:9001"
    skus:
      - "a1b2c3d4-e5f6-7890-abcd-1234567890ab"
      - "b2c3d4e5-f6a7-8901-bcde-2345678901bc"
    default: true

  - name: docker-2
    url: "http://10.0.0.2:9001"
    skus:
      - "a1b2c3d4-e5f6-7890-abcd-1234567890ab"
      - "b2c3d4e5-f6a7-8901-bcde-2345678901bc"

callback_base_url: "http://fred.provider.example.com:8080"
callback_secret: "your-32-character-or-longer-secret-here"

# Required for multi-backend setups (multiple backends sharing match criteria).
# Records which backend serves each lease so reads hit the right machine.
# placement_store_db_path: "/var/lib/fred/placements.db"

Per-backend fields:

Validation rules:

• Backend names must be unique
• Backend URLs must be absolute http:// or https:// URLs with a host
• callback_base_url must be an absolute http:// or https:// URL
• Trailing slashes on callback_base_url are automatically stripped
• callback_canonical_path_prefix, when set, must start with / and must not end with /

Note: The Docker backend has additional configuration (releases_db_path, releases_max_age, container_stop_timeout, etc.) documented in docker-backend.example.yaml.

These options have sensible defaults but can be tuned for specific environments:

See SECURITY.md for TLS configuration details (API server HTTPS, gRPC to chain).

All options can be set via environment variables with the PROVIDER_ prefix:

export PROVIDER_CHAIN_ID=manifest-1
export PROVIDER_PROVIDER_UUID=01234567-89ab-cdef-0123-456789abcdef
export PROVIDER_CALLBACK_BASE_URL=http://fred.example.com:8080

# Run with config file
./build/providerd -c config.yaml

# Or use environment variables
./build/providerd

# Print version (providerd, docker-backend, and k3s-backend support --version)
./build/providerd --version
./build/docker-backend --version
./build/k3s-backend --version

k3s-backend is currently an experimental, non-functional scaffold (ENG-133): the binary boots, serves the backend HTTP contract, and signs/verifies callbacks, but its provisioner returns status=failed, error="not implemented" for every provision. It is not usable in production; real Kubernetes provisioning lands in ENG-134+.

See SECURITY.md for replay protection rationale per endpoint.

GET /health

Returns server health status. Checks chain connectivity, all registered backends, token tracker (bbolt), and placement store (bbolt). Returns 200 OK when all checks pass or 503 Service Unavailable when any check fails.

Response:

{
  "status": "healthy",
  "provider_uuid": "01234567-89ab-cdef-0123-456789abcdef",
  "checks": {
    "chain": {"status": "healthy"},
    "backend:docker-1": {"status": "healthy"},
    "token_tracker": {"status": "healthy"},
    "placement_store": {"status": "healthy"}
  }
}

GET /v1/leases/{lease_uuid}/connection
Authorization: Bearer <token>

Returns connection details for an active lease from the backend. Requires ADR-036 signed bearer token. See SECURITY.md for token format and signing details.

Response (single instance):

{
  "lease_uuid": "...",
  "tenant": "manifest1...",
  "provider_uuid": "...",
  "connection": {
    "host": "compute-alpha.example.com",
    "fqdn": "a1b2c3d.example.com",
    "ports": {
      "8080/tcp": {"host_ip": "0.0.0.0", "host_port": 32768},
      "443/tcp": {"host_ip": "0.0.0.0", "host_port": 32769}
    },
    "protocol": "https",
    "metadata": {
      "region": "us-east-1",
      "backend": "kubernetes"
    }
  }
}

Response (multi-instance lease):

{
  "lease_uuid": "...",
  "tenant": "manifest1...",
  "provider_uuid": "...",
  "connection": {
    "host": "compute-alpha.example.com",
    "fqdn": "0-a1b2c3d.example.com",
    "instances": [
      {
        "instance_index": 0,
        "container_id": "abc123",
        "image": "nginx:latest",
        "status": "running",
        "fqdn": "0-a1b2c3d.example.com",
        "ports": {"80/tcp": {"host_ip": "0.0.0.0", "host_port": 32768}}
      },
      {
        "instance_index": 1,
        "container_id": "def456",
        "image": "redis:alpine",
        "status": "running",
        "fqdn": "1-e5f6789.example.com",
        "ports": {"6379/tcp": {"host_ip": "0.0.0.0", "host_port": 32769}}
      }
    ],
    "metadata": {"backend": "docker"}
  }
}

Fields:

• fqdn - Fully qualified domain name for ingress routing (omitted when ingress is not enabled). At the top level (connection.fqdn), this is set directly from the backend or propagated from the first instance's FQDN when no top-level value is provided. Each instance and service may also have its own fqdn. A top-level or service-level explicit FQDN takes precedence over instance propagation.
• ports - Map of container port to host binding (e.g., "8080/tcp" → host_port 32768)
• instances - Array of per-instance details for multi-container leases (each with its own ports and optional fqdn)
• services - Map of service name to connection details for stack (multi-service) leases. Each service contains its own instances array and optional fqdn (propagated from its first instance when not set explicitly).
• metadata - Additional backend-specific data

GET /v1/leases/{lease_uuid}/status
Authorization: Bearer <token>

Returns the current provisioning status of a lease. Useful for checking if provisioning is in progress or complete.

Response:

{
  "lease_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "tenant": "manifest1abc...",
  "provider_uuid": "01234567-89ab-cdef-0123-456789abcdef",
  "state": "PENDING",
  "requires_payload": true,
  "meta_hash_hex": "a1b2c3...",
  "payload_received": false,
  "provisioning_started": false
}

Fields:

• tenant - Tenant address from the authenticated token
• provider_uuid - Provider UUID
• state - Chain lease state (PENDING, ACTIVE, CLOSED, EXPIRED)
• requires_payload - True if lease has meta_hash (expects payload upload)
• meta_hash_hex - Expected payload hash in hex (omitted if no meta_hash)
• payload_received - True if payload has been uploaded
• provisioning_started - True if provisioning is in progress
• provision_status - Backend provision status (omitted if not provisioned). May be retained for a closed/expired lease whose data was soft-deleted and is restorable (see retention)
• fail_count - Number of provisioning failures (omitted if zero)
• reason - Stable, machine-readable failure category; present whenever a failure has been recorded — including a ready lease whose last update failed and rolled back to the previous version — and omitted (omitempty) when empty; see Failure Reason Codes
• message - Curated, human-readable failure summary (omitted if empty); no host paths or raw command output
• retained_until - RFC3339 grace-window deadline; present only when provision_status is retained
• items - Restore shape (service_name, sku, quantity) to request when opening the fresh lease to restore into; present only when retained
• restore_hint - Short human-readable next step for restoring; present only when retained

Chain-pruned leases: after a lease is auto-closed and pruned from the chain, this endpoint still answers from the retained record. In that case authorization is by the retained record's tenant (the signed caller must own it); a cross-tenant caller or an absent record gets 404.

GET /v1/leases/{lease_uuid}/provision
Authorization: Bearer <token>

Returns provision diagnostics for a lease, including status, failure reason, and failure count. Works for both active and non-active leases (e.g., after rejection or closure), falling back to persisted diagnostics when the provision is no longer in memory.

Response:

{
  "lease_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "tenant": "manifest1abc...",
  "provider_uuid": "01234567-89ab-cdef-0123-456789abcdef",
  "status": "failed",
  "fail_count": 3,
  "reason": "ContainerExited",
  "message": "container exited unexpectedly"
}

Fields:

• status - Provision status: provisioning, ready, failing, failed, restarting, updating, deprovisioning, retained, or unknown. failing is a transient state between container-death detection and the Failed callback; deprovisioning covers the container-removal window; retained marks a closed/expired lease whose data was soft-deleted and is restorable
• fail_count - Number of provision attempts that failed
• reason - Stable, machine-readable failure category, always present when status is failed (defaults to Unknown if no specific cause was recorded); see Failure Reason Codes
• message - Curated, human-readable failure summary; may be empty
• retained_until, items, restore_hint - Present only when status is retained (grace-window deadline, restore shape, and next-step hint); see Get Lease Status

Response Codes:

• 200 OK - Provision found
• 401 Unauthorized - Invalid signature or token
• 403 Forbidden - Lease does not belong to this tenant
• 404 Not Found - Provision not found (never provisioned or diagnostics expired)

reason is an open, add-only enum (Kubernetes Condition.Reason-shaped): new values may be added in future releases without notice. Clients must treat any value they don't recognize as a generic failure and fall back to displaying the human-readable message — never match/branch on reason with an exhaustive switch that errors on the default case. Adding a new reason is considered a backward-compatible change.

The set defined today:

message is a short, human-readable string for display; it contains no host filesystem paths or raw command/daemon output (that detail is retained operator-side, correlated by lease_uuid, for support/debugging).

GET /v1/leases/{lease_uuid}/logs?tail=100
Authorization: Bearer <token>

Returns container logs for a lease. Works for both active and non-active leases, falling back to persisted logs when the provision is no longer in memory.

Query Parameters:

• tail - Number of log lines to return per container (default: 100, max: 10000)

Response:

{
  "lease_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "tenant": "manifest1abc...",
  "provider_uuid": "01234567-89ab-cdef-0123-456789abcdef",
  "logs": {
    "0": "2024-01-15 Starting nginx...\nListening on port 80\n",
    "1": "2024-01-15 Redis ready\n"
  }
}

Fields:

• logs - Map of container instance index to log output

Response Codes:

• 200 OK - Logs found
• 400 Bad Request - Invalid tail parameter (negative, zero, or exceeds max)
• 401 Unauthorized - Invalid signature or token
• 403 Forbidden - Lease does not belong to this tenant
• 404 Not Found - Provision not found (never provisioned or logs expired)

The payload is a deployment manifest in JSON format. See the Manifest Guide for the full schema (single-service and stack formats, validation rules, examples). A formal JSON Schema is available for client-side validation.

POST /v1/leases/{lease_uuid}/data
Authorization: Bearer <token>
Content-Type: application/octet-stream

<raw payload bytes>

Upload deployment configuration for a lease that was created with a meta_hash. The payload is validated against the on-chain hash before provisioning starts. Requires a payload-specific ADR-036 token that includes the meta_hash field. See SECURITY.md for token details.

Response Codes:

• 202 Accepted - Payload received, provisioning started
• 400 Bad Request - Invalid payload or hash mismatch
• 401 Unauthorized - Invalid signature or token
• 404 Not Found - Lease not found or not PENDING
• 409 Conflict - Payload already received

POST /v1/leases/{lease_uuid}/restart
Authorization: Bearer <token>

Restart containers for a lease without changing the manifest. Containers are stopped, removed, and recreated with the same configuration. Volumes are preserved. Allowed from ready or failed state.

Response: 202 Accepted

{
  "status": "restarting"
}

Response Codes:

• 202 Accepted - Restart initiated
• 401 Unauthorized - Invalid signature or token
• 403 Forbidden - Lease does not belong to this tenant
• 404 Not Found - Lease not provisioned
• 409 Conflict - Lease is in a state that cannot be restarted (e.g., already restarting or updating)

POST /v1/leases/{lease_uuid}/update
Authorization: Bearer <token>
Content-Type: application/json

{
  "payload": "<base64-encoded-manifest>"
}

Deploy a new manifest for a lease, replacing containers with a new image/configuration. The old containers are stopped, new ones are created from the updated manifest, and old containers are cleaned up after verification. On failure, the operation rolls back to the previous containers. Volumes are preserved.

Response: 202 Accepted

{
  "status": "updating"
}

Response Codes:

• 202 Accepted - Update initiated
• 400 Bad Request - Invalid payload or manifest validation error
• 401 Unauthorized - Invalid signature or token
• 403 Forbidden - Lease does not belong to this tenant
• 404 Not Found - Lease not provisioned
• 409 Conflict - Lease is in a state that cannot be updated (e.g., currently restarting)

POST /v1/leases/{lease_uuid}/restore
Authorization: Bearer <token>
Content-Type: application/json

{
  "from_lease_uuid": "<original-closed-lease-uuid>"
}

Restore a soft-deleted lease's retained data into a new lease. The path lease_uuid is the new, fresh PENDING lease the data is adopted into; from_lease_uuid in the body names the original closed/expired lease whose volumes were retained (see retention). Fred resolves the backend that holds the source lease's retained data (restore is same-backend, ENG-333), then re-deploys the retained manifest onto the adopted volumes. Only the item shape must match: the new lease's requested service names and quantities must equal the original's, but its SKU/disk tier MAY differ. A promote (same-or-larger disk tier) is always allowed and the new disk_mb cap is applied; a demote (smaller disk tier) is allowed only if the retained volume's measured data still fits the new tier's disk_mb cap (the backend runs checkDemoteFit before adopting). A refused demote returns 422 Unprocessable Entity; the JSON body's error message begins retained data exceeds the requested smaller tier (the body's code field is the numeric HTTP status, not a string discriminator).

Response: 202 Accepted

{
  "status": "provisioning"
}

Response Codes:

• 202 Accepted - Restore initiated (the lease then transitions through restarting to ready/failed)
• 400 Bad Request - Missing/invalid from_lease_uuid, or items don't match the retained set
• 401 Unauthorized - Invalid signature or token
• 403 Forbidden - Lease does not belong to this tenant
• 404 Not Found - No retained data found for from_lease_uuid (absent, expired, cross-tenant, or its backend is gone)
• 409 Conflict - Target lease is not PENDING, is already provisioned, or is not in a restorable state
• 422 Unprocessable Entity - Requested a smaller SKU tier (demote) but the retained data exceeds the new tier's disk_mb cap; the error message begins retained data exceeds the requested smaller tier
• 503 Service Unavailable - Insufficient resources to restore, or placement routing is not configured

GET /v1/leases/{lease_uuid}/releases
Authorization: Bearer <token>

Returns the release (deployment) history for a lease, showing each version that was deployed.

Response:

{
  "lease_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "tenant": "manifest1abc...",
  "provider_uuid": "01234567-89ab-cdef-0123-456789abcdef",
  "releases": [
    {
      "version": 1,
      "image": "nginx:1.24",
      "status": "superseded",
      "created_at": "2024-01-15T10:30:00Z",
      "manifest": "<base64-encoded-manifest>"
    },
    {
      "version": 2,
      "image": "nginx:1.25",
      "status": "active",
      "created_at": "2024-01-16T14:00:00Z",
      "manifest": "<base64-encoded-manifest>"
    }
  ]
}

Fields:

• version - Monotonically increasing version number
• image - Container image used in this release
• status - Release status: deploying, active, superseded, or failed
• created_at - When this release was created
• reason - Stable, machine-readable failure category (only present on failed releases); see Failure Reason Codes
• message - Curated, human-readable failure summary (only present on failed releases; may be empty)
• manifest - The manifest payload used for this release

Response Codes:

• 200 OK - Releases found (may be an empty array)
• 401 Unauthorized - Invalid signature or token
• 403 Forbidden - Lease does not belong to this tenant
• 404 Not Found - Lease not provisioned

GET /v1/leases/{lease_uuid}/events
Authorization: Bearer <token>

Opens a WebSocket connection for real-time lease status updates. Events are pushed as JSON frames when the lease transitions between provisioning states (e.g., provisioning, ready, failed, restarting, updating). A retained event is pushed when a closed/expired lease's data is soft-deleted (best-effort, only to currently-connected clients), signalling that the data may be restorable within the grace window. For that event the status field is the enum retained, and the human-readable restore instruction is carried in the error field.

Authentication: Bearer token via the Authorization header or the ?token= query parameter (since the WebSocket API cannot set custom headers during upgrade). Auth is verified before the WebSocket upgrade, so failures return standard HTTP error responses.

Response: 101 Switching Protocols on successful upgrade

{"lease_uuid":"...","status":"ready","timestamp":"2024-01-15T10:30:00Z"}
{"lease_uuid":"...","status":"restarting","timestamp":"2024-01-15T10:31:00Z"}
{"lease_uuid":"...","status":"ready","timestamp":"2024-01-15T10:31:30Z"}

Behavior:

• Events are delivered as WebSocket JSON frames
• The server sends WebSocket ping frames every 30 seconds; the client must respond with pong within 40 seconds or the connection is closed
• Slow clients that fall behind have events dropped — use the REST endpoints (/status, /releases) to catch up
• The stream ends when the client disconnects or the server shuts down (clean close frame)

Response Codes (before upgrade):

• 101 Switching Protocols - WebSocket connection established
• 401 Unauthorized - Invalid signature or token
• 403 Forbidden - Lease does not belong to this tenant
• 501 Not Implemented - Events not enabled on this deployment

POST /callbacks/provision
Content-Type: application/json
X-Fred-Signature: t=<unix-timestamp>,sha256=<hmac-sha256-hex>

Called by backends to report provisioning status. Requires HMAC-SHA256 authentication via the X-Fred-Signature header. See SECURITY.md for signing details and replay protection.

Request:

{
  "lease_uuid": "...",
  "status": "success",
  "error": "",
  "retained": false
}

Status must be one of "success", "failed", or "deprovisioned" (the third is used by backends that perform autonomous deprovisioning, e.g. after a failed provision rollback).

• retained (optional bool) — set true on a deprovisioned callback when the backend soft-deleted (retained) the lease's volumes instead of destroying them. Fred uses this to push the optimistic retained notice to the tenant; the queryable retained status (GET /v1/leases/{uuid}/status) is the durable backstop. Omitted/false means the volumes were destroyed.

Response Codes:

• 200 OK - Callback processed successfully (or already processed)
• 401 Unauthorized - Missing or invalid signature

Idempotency: If a callback is received for a lease that has already been processed (no longer in-flight), the server still returns 200 OK as a no-op. Response bodies are not guaranteed for this path, so callers should treat the HTTP status code as the source of truth.

Any backend must implement these HTTP endpoints. For a comprehensive implementation guide including SKU handling, callback signing, state management, and reconciliation, see BACKEND_GUIDE.md.

All endpoints except /health, /stats, and /metrics require HMAC-SHA256 signature authentication via the X-Fred-Signature header.

Backends without soft-delete/retention support still serve /restore and /retentions: /restore returns 422 (no retained data) and /retentions returns an empty list.

Start provisioning a resource (async).

Request:

{
  "lease_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "tenant": "manifest1abc...",
  "provider_uuid": "01234567-89ab-cdef-0123-456789abcdef",
  "items": [
    {"sku": "k8s-small", "quantity": 2},
    {"sku": "k8s-large", "quantity": 1}
  ],
  "callback_url": "http://fred.example.com:8080/callbacks/provision",
  "payload": "<base64-encoded-bytes>",
  "payload_hash": "abc123..."
}

Fields:

• items - Array of lease items with SKU and quantity. All items belong to the same provider.
• payload - Optional base64-encoded deployment payload (only present if lease has meta_hash)
• payload_hash - Optional hex-encoded SHA-256 hash of payload (only present with payload)

Response: 202 Accepted

{
  "provision_id": "..."
}

Get lease information for a provisioned resource.

Response: 200 OK

{
  "host": "10.0.0.1",
  "ports": {
    "8080/tcp": {"host_ip": "0.0.0.0", "host_port": "32768"},
    "443/tcp": {"host_ip": "0.0.0.0", "host_port": "32769"}
  },
  "protocol": "https",
  "metadata": {"region": "us-east-1"},
  "custom_field": "any additional backend-specific data"
}

Known Fields (extracted by fred into structured response):

• host - Hostname or IP for connecting to the resource
• fqdn - Fully qualified domain name for ingress routing (omitted when not set)
• ports - Map of container ports to host bindings
• instances - Array of per-instance details (each may include its own fqdn)
• services - Map of service name to per-service details (each may include its own fqdn)
• protocol - Connection protocol (e.g., "https", "ssh")
• metadata - Additional key-value metadata

Backends should use the metadata field for any custom key-value data to surface to tenants.

Response: 404 Not Found if not provisioned.

Deprovision a resource (idempotent).

Request:

{
  "lease_uuid": "550e8400-e29b-41d4-a716-446655440000"
}

Response: 200 OK

Get provision diagnostics for a specific lease.

Response: 200 OK

{
  "lease_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "provider_uuid": "01234567-89ab-cdef-0123-456789abcdef",
  "status": "failed",
  "fail_count": 3,
  "reason": "ContainerExited",
  "message": "container exited unexpectedly",
  "created_at": "2024-01-15T10:30:00Z"
}

Response: 404 Not Found if not provisioned.

Get container logs for a specific lease.

Query Parameters:

• tail - Number of log lines per container (default: 100)

Response: 200 OK

{
  "0": "2024-01-15 Starting nginx...\nListening on port 80\n",
  "1": "2024-01-15 Redis ready\n"
}

Response: 404 Not Found if not provisioned.

List all provisions (for reconciliation).

GET /provisions is keyset-paginated. Query params: limit (max page size) and continue (a lease UUID — the continue cursor returned by the previous page). The JSON response carries a top-level continue field set to the last record's lease UUID, omitted once the list is exhausted. An invalid limit or a non-UUID continue returns 400, as does a continue cursor supplied without a positive limit. A limit above the server maximum (5000) is coerced down to it rather than rejected. With no params it returns the full list unpaginated (back-compat). One or more lease_uuid query params return just those records. (ENG-380)

Response:

{
  "provisions": [
    {
      "lease_uuid": "...",
      "status": "ready",
      "created_at": "2024-01-15T10:30:00Z"
    }
  ],
  "continue": "..."
}

Restart containers for a lease without changing the manifest (async).

Request:

{
  "lease_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "callback_url": "http://fred.example.com:8080/callbacks/provision"
}

Response: 202 Accepted

{
  "status": "restarting"
}

Error Responses:

• 404 Not Found - Lease not provisioned
• 409 Conflict - Invalid state for restart (e.g., already restarting or updating)

Deploy a new manifest for a lease, replacing containers (async).

Request:

{
  "lease_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "callback_url": "http://fred.example.com:8080/callbacks/provision",
  "payload": "<base64-encoded-manifest>",
  "payload_hash": "sha256-hex-string"
}

Response: 202 Accepted

{
  "status": "updating"
}

Error Responses:

• 400 Bad Request - Invalid manifest or validation error
• 404 Not Found - Lease not provisioned
• 409 Conflict - Invalid state for update

Adopt a soft-deleted lease's retained volumes into a new lease and re-deploy its retained manifest (async). lease_uuid is the new lease; from_lease_uuid is the original retained lease. items must shape-match the retained set.

Request:

{
  "lease_uuid": "<new-lease-uuid>",
  "from_lease_uuid": "<original-retained-lease-uuid>",
  "tenant": "manifest1abc...",
  "provider_uuid": "01234567-89ab-cdef-0123-456789abcdef",
  "items": [{"sku": "docker-redis", "quantity": 1, "service_name": "app"}],
  "callback_url": "http://fred.example.com:8080/callbacks/provision"
}

Response: 202 Accepted

{
  "status": "restoring"
}

Error Responses:

• 400 Bad Request - Missing lease_uuid/from_lease_uuid/callback_url, or items/manifest validation error
• 409 Conflict - Invalid state for restore, or already provisioned. Both return a JSON {"error": "..."} body; the already-provisioned case additionally sets code: "already_provisioned", so the two are distinguished by the presence of that discriminator
• 422 Unprocessable Entity - No retained data for the source lease (also returned by backends that don't support retention)
• 503 Service Unavailable - Insufficient resources

Fred maps only the backend's bare 422 (ErrNotRetained — no retained data, no code) to a tenant-facing 404 on POST /v1/leases/{uuid}/restore. A 422 carrying code: "demote_exceeds_tier" (ErrDemoteDataExceedsTier — retained data exceeds the requested smaller tier) is forwarded to the tenant as 422, not remapped.

List the leases whose data this backend currently retains (soft-deleted, awaiting restore or grace-reap). Fred's reconciler polls this on every backend to keep restore routing affinity (a restore is routed to the backend that holds the source data). Backends without retention return an empty list.

Response: 200 OK

{
  "retentions": [
    {"lease_uuid": "550e8400-e29b-41d4-a716-446655440000"}
  ]
}

The retentions array is always present ([] when empty, never null).

Get release (deployment) history for a lease.

Response: 200 OK

[
  {
    "version": 1,
    "image": "nginx:1.24",
    "status": "superseded",
    "created_at": "2024-01-15T10:30:00Z",
    "manifest": "<base64-encoded-manifest>"
  },
  {
    "version": 2,
    "image": "nginx:1.25",
    "status": "active",
    "created_at": "2024-01-16T14:00:00Z",
    "manifest": "<base64-encoded-manifest>"
  }
]

Response: 404 Not Found if not provisioned.

The mock backend allows you to test fred's provisioning flow without a real backend. It supports concurrent provisions with per-lease callback routing.

Note: The mock backend ignores the SKU field entirely - all provisions create identical fake resources regardless of SKU. Connection details are deterministically generated from the lease UUID. For implementing a real backend that interprets SKUs, see BACKEND_GUIDE.md.

# Build mock-backend
make build-mock

# Run with required callback secret
MOCK_BACKEND_CALLBACK_SECRET="test-secret-at-least-32-characters-long" ./build/mock-backend

# Or with custom settings
MOCK_BACKEND_ADDR=:9001 \
MOCK_BACKEND_NAME=test-backend \
MOCK_BACKEND_DELAY=2s \
MOCK_BACKEND_CALLBACK_SECRET="test-secret-at-least-32-characters-long" \
./build/mock-backend

Environment Variables:

Note: The mock backend stores callback URLs per lease UUID, so concurrent provisions with different callback URLs are handled correctly without race conditions.

Security Warning: The mock backend accepts arbitrary callback_url values and issues HTTP requests to them, which is an SSRF risk if exposed to untrusted networks. Only run the mock backend on trusted interfaces (e.g., localhost) for local testing. Do not expose it to the internet or untrusted users.

Create a config file that points to the mock backend:

# config-test.yaml
provider_uuid: "01234567-89ab-cdef-0123-456789abcdef"
provider_address: "manifest1test..."
keyring_backend: "test"
keyring_dir: "/tmp/test-keyring"
key_name: "test"

chain_id: "test-chain"
grpc_endpoint: "localhost:9090"
websocket_url: "ws://localhost:26657/websocket"

api_listen_addr: ":8080"

backends:
  - name: mock
    url: "http://localhost:9000"
    timeout: 30s
    default: true

callback_base_url: "http://localhost:8080"
callback_secret: "test-secret-at-least-32-characters-long"

./build/providerd -c config-test.yaml

Check mock backend health:

curl http://localhost:9000/health

Simulate a provision request (directly to mock backend):

curl -X POST http://localhost:9000/provision \
  -H "Content-Type: application/json" \
  -d '{
    "lease_uuid": "550e8400-e29b-41d4-a716-446655440000",
    "tenant": "manifest1abc",
    "provider_uuid": "01234567-89ab-cdef-0123-456789abcdef",
    "items": [{"sku": "mock-resource", "quantity": 1}],
    "callback_url": "http://localhost:8080/callbacks/provision"
  }'

Check provisioned resources:

curl http://localhost:9000/provisions

Get lease info:

curl http://localhost:9000/info/550e8400-e29b-41d4-a716-446655440000

Deprovision:

curl -X POST http://localhost:9000/deprovision \
  -H "Content-Type: application/json" \
  -d '{"lease_uuid": "550e8400-e29b-41d4-a716-446655440000"}'

Create a docker-compose.yaml for integrated testing:

version: '3.8'

services:
  mock-backend:
    # Build once on the host first: make build-mock
    image: alpine:3.21
    command: ["/app/mock-backend"]
    environment:
      - MOCK_BACKEND_ADDR=:9000
      - MOCK_BACKEND_DELAY=1s
      - MOCK_BACKEND_CALLBACK_SECRET=shared-secret-at-least-32-characters
    volumes:
      - "./build/mock-backend:/app/mock-backend:ro"
    ports:
      - "9000:9000"

  fred:
    build:
      context: .
      dockerfile: Dockerfile
      target: providerd
    environment:
      - PROVIDER_PROVIDER_UUID=01234567-89ab-cdef-0123-456789abcdef
      - PROVIDER_API_LISTEN_ADDR=:8080
      - PROVIDER_CALLBACK_BASE_URL=http://fred:8080
      - PROVIDER_CALLBACK_SECRET=shared-secret-at-least-32-characters
    ports:
      - "8080:8080"
    depends_on:
      - mock-backend

cmd/
├── providerd/          # Main daemon entry point
├── mock-backend/       # Mock backend for testing
├── docker-backend/     # Docker container backend
├── k3s-backend/        # K3s container backend
├── lease-token/        # Mints ADR-036 tenant bearer tokens for lease endpoints
└── loadtest/           # Load testing tool (not built by `make all`; `go build ./cmd/loadtest`)

internal/
├── adr036/             # ADR-036 signature verification
├── api/                # HTTP server, handlers, rate limiting
├── auth/               # Shared authentication utilities
├── hmacauth/           # HMAC-SHA256 signing and verification
├── backend/            # Backend client and router
│   ├── client.go       # HTTP client for backends (with circuit breaker)
│   ├── router.go       # SKU-based routing
│   ├── mock.go         # In-memory mock for unit tests
│   ├── shared/         # Cross-backend primitives (callback sender, bbolt store, registry, diagnostics)
│   ├── docker/         # Docker container backend implementation (actor-per-lease)
│   └── k3s/            # K3s container backend implementation
├── chain/              # gRPC client, WebSocket subscriber, signer
│   └── chaintest/      # Test-only mock chain client (not imported by providerd)
├── config/             # Configuration loading and validation
├── metrics/            # Prometheus metrics definitions
├── provisioner/        # Provision lifecycle management
│   ├── manager.go      # Coordinator (wires components together)
│   ├── orchestrator.go # Routes to backends, starts provisioning
│   ├── handlers.go     # Shared handler logic and lease item extraction
│   ├── handler_set.go  # Watermill message handlers
│   ├── reconciler.go   # Level-triggered state reconciliation
│   ├── tracker.go      # InFlightTracker interface + DefaultInFlightTracker implementation
│   ├── inflight.go     # Manager delegation methods to the tracker
│   ├── ack_batcher.go  # Batches lease acknowledgments
│   ├── timeout_checker.go # Detects callback timeouts
│   ├── leaseutil.go    # Lease helper utilities
│   ├── topics.go       # Watermill topic name constants
│   ├── payload/        # Temporary payload storage (bbolt)
│   ├── placement/      # Lease→backend placement store (bbolt)
│   ├── bridge.go       # Chain events -> Watermill
│   └── interfaces.go   # BackendRouter, LeaseRejecter, PlacementStore interfaces
├── scheduler/          # Periodic withdrawal and credit monitoring
├── testutil/           # Test fixtures and helpers
├── tlsconfig/          # TLS config builders for the providerd<->backend hop (mTLS, identity pinning)
├── util/               # Shared utility functions
└── watcher/            # Cross-provider event detection

Fred uses level-triggered reconciliation to ensure consistency between chain state and backend state. This provides crash recovery without requiring durable event queues.

Instead of replaying missed events (edge-triggered), reconciliation queries current state. Before reading provisions, the reconciler calls RefreshState on each backend to ensure in-memory state is synchronized with the actual infrastructure (e.g., Docker container status).

Chain State (leases)     Backend State (provisions)
        │                          │
        └──────────┬───────────────┘
                   │
                   ▼
         RefreshState (each backend)
                   │
                   ▼
            Reconciler compares
                   │
        ┌──────────┼──────────┬──────────┐
        ▼          ▼          ▼          ▼
    PENDING     ACTIVE      ACTIVE     CLOSED
    + not      + not       + failed   + still
    provisioned provisioned provision  provisioned
        │          │          │          │
        ▼          ▼          ▼          ▼
     Start      Anomaly:  Re-provision Deprovision
   provisioning  log &    (with limit)  (orphan
                provision               cleanup)

1. Startup: Full reconciliation runs immediately on startup
2. Periodic: Runs every reconciliation_interval (default: 5 minutes)
3. Cross-provider credit depletion: Triggers withdrawal which may close leases

The last row takes precedence over every other. A sweep applies the matrix only to leases whose owning backend reported: one present in the backend data, or one whose placement record names a backend that answered. Anything else is deferred and retried on the next sweep, because acting on a lease fred cannot see risks re-provisioning it onto a healthy peer and laying an empty volume over live data.

A backend failing to answer therefore degrades only its own leases; it no longer stops reconciliation for the rest of the fleet. Sweeps that ran degraded are reported by fred_reconciler_sweep_complete (0) and counted as fred_reconciler_runs_total{outcome="degraded"}.

The three passes that delete durable state — orphan deprovision, payload cleanup, placement pruning — keep running on a degraded sweep, scoped to what that sweep can positively account for:

Absence is never evidence. The two lease-list queries are filtered to PENDING/ACTIVE and are not atomic, so "missing from the sweep" means terminal or never-known or created seconds ago; and because the ledger never deletes a lease, a chain with no record of one means a phantom provision, a wrong or reset chain, or a lagging RPC node. A failed re-read is likewise not absence. Every such case keeps the state and increments fred_reconciler_cleanup_skips_total{pass,reason}.

• Tenant Authentication: ADR-036 secp256k1 signatures with 30-second token expiry and low-S normalization
• Replay Protection: Persistent token tracking (bbolt) with fail-closed semantics on mutating endpoints
• Callback Authentication: HMAC-SHA256 with timestamp-based replay protection (5-minute window)
• Rate Limiting: Dual-layer token bucket — one per-IP limiter shared across all routes (10 RPS) and a per-tenant limiter (5 RPS); behind a proxy, set trusted_proxies so it keys on the real client IP
• Container Hardening: Drop all capabilities, no-new-privileges, read-only rootfs, PID limits, network isolation
• Input Validation: UUID format checks, URL scheme/host validation, manifest parsing, image allowlisting
• Production Mode: Enforces replay protection, blocks TLS skip-verify, SSRF checks on all URLs
• Constant-Time Comparisons: hmac.Equal and subtle.ConstantTimeCompare for all secret comparisons

See SECURITY.md for the full security architecture, authentication flows, replay protection rationale, and known limitations.

Fred's event processing pipeline has been extensively benchmarked:

See PERFORMANCE.md for detailed benchmarks, stress test results, and comparison with other solutions.

• Go 1.26.5+ (per the go 1.26.5 directive in go.mod; also uses sync.WaitGroup.Go(), testing.B.Loop(), range over integers)
• Watermill (event routing)
• Cosmos SDK v0.50.14
• CometBFT v0.38.x
• manifest-ledger (for billing/sku types)

Licensed under the Apache License, Version 2.0. See LICENSE for the full text.