Sync never delivers the first entry to a reader connected only through a relay
Package: @orbitdb/core@4.0.0 (src/sync.js)
Summary
In a star topology where two peers can reach each other only through a third
(relay/pinner) — the common browsers-behind-NAT case — a reader can be
permanently unable to receive the writer's first database entry, even though
both peers are connected to the relay, subscribed to the database topic, and
grafted into the relay's gossipsub mesh. The data sits in the relay's OrbitDB
and is never delivered onward.
The root cause is that Sync couples "receive an entry" and "announce an entry"
too tightly to the pubsub publish path and to the one-shot
subscription-change edge:
sync.add() publishes only locally-appended entries (sync.js:257).
Entries obtained via the heads-exchange protocol are never re-announced on
the topic — so a middleman that learns an entry by exchange never forwards
it via pubsub.
- The heads exchange is one-shot per peer (
sync.js:192,
peers.has(peerId) → return). If a peer exchanges heads while the other side
is still empty, it never re-exchanges when heads appear later (only a
peer:disconnect clears the cache, sync.js:246).
- Peer discovery is edge-triggered (
sync.js:291): startSync only reacts
to subscription-change events after attaching its listener; a peer that
was already subscribed when the database opens never triggers an exchange.
Individually each is defensible; together they deadlock the topology above.
What happens
Writer A, reader B, relay R. A—R and B—R only (A and B never meet).
- A publishes its first entry. If R is not subscribed to the topic at that
instant (or A's mesh/fanout has no subscribed target yet), the single
pubsub.publish reaches nobody. There is no retry and no later re-announce.
- R later obtains the entry via the heads-exchange protocol with A (not via
pubsub). Per (1), R does not publish it onward.
- B exchanged heads with R earlier, while R was still empty. Per (2), B is in
R's one-shot peers cache and is never re-exchanged with. Per (3), nothing
re-triggers it.
B is connected, subscribed and mesh-grafted — and never receives the entry.
Reproduction
Deterministic, local, ~20 s, only @orbitdb/core + helia + libp2p:
Permalink (runnable mocha spec): https://github.com/NiKrause/orbitdb-relay/blob/1f393a1/mocha/upstream-sync-preexisting-subscribers.mjs
The construction forces the end-state without depending on scheduler timing:
open the database on all three nodes and settle a gossipsub mesh (the relay
grafts both peers); isolate the writer and append the first entry so its
publish reaches nobody; let the writer's gossipsub message cache (mcache)
expire so a later reconnect cannot rescue the reader via IHAVE/IWANT;
reconnect the writer so the relay obtains the entry via the heads exchange
(not a publish); assert the reader stays empty (the bug); then a single
dbRelay.sync.add(head) delivers it — proving a Sync coordination gap, not a
transport fault.
3/3 runs green (green = bug reproduced).
Why local reproduction needs the isolation/mcache-expiry steps: on a fast
local network the relay subscribes and the mesh grafts within ~200 ms, so the
writer's publish almost always lands after the relay is ready and the entry is
delivered normally. The deadlock surfaces naturally only when mesh/subscription
formation is slow relative to the first publish — which is exactly what a real
deployment does. See the field evidence below.
Field evidence
Observed on a live relay (browser on a CI runner + browser on a cloud VM,
connected only via a WebSocket/circuit-relay pinning node), gossipsub debug on
the relay, on a brand-new database topic T:
13:59:32.9 writer publishes the first entry
13:59:33.1 relay JOINs T, adds mesh links to both peers
13:59:34.0 GRAFT from reader → reader in relay mesh
13:59:36.0 GRAFT from writer → mesh only fully formed here, ~3 s after the publish
… relay obtains the entry via the heads exchange with the writer …
… reader never receives it; test times out at 120 s …
The mesh does form — ~3 s too late for the first publish — and nothing
re-announces the entry afterward.
Suggested fixes (any one breaks the deadlock)
startSync: enumerate already-subscribed peers. Call the same
join/exchange path for pubsub.getSubscribers(address) at start, instead of
relying only on the subscription-change edge.
- Re-exchange / re-announce when local heads change after receiving heads
(or expose a supported "request re-exchange" that does not require
stop()/start(), which also tears down the subscription).
- Allow a middleman to re-announce exchanged heads on the topic, so a
pinner that learns an entry by exchange forwards it via pubsub.
Workaround we ship
In our relay: after a sync pass that received updates, re-announce the current
heads via db.sync.add(head), so every current subscriber receives them
regardless of exchange ordering (receivers de-duplicate via joinEntry).
Note: keeping the relay pubsub-subscribed across database closes does not
work — because discovery is edge-triggered, a standing subscription turns the
next open into a no-op change and no peer ever dials the exchange (we reproduced
this as a hard regression).
Full reproduction test (self-contained mocha spec)
// Minimal reproduction of an @orbitdb/core Sync deadlock, intended for an
// upstream issue. Uses only @orbitdb/core + helia + libp2p — none of this
// repository's relay code.
//
// The defect (sync.js, @orbitdb/core 4.0.0):
// 1. `startSync()` discovers peers ONLY via `subscription-change` events —
// peers already subscribed when the database opens are never exchanged
// with.
// 2. `peers` is a one-shot cache: each peer gets exactly one heads
// exchange until disconnect; there is no re-exchange when local heads
// change later.
// 3. `sync.add()` publishes only locally-appended entries; heads received
// via the exchange are never re-announced on the topic.
//
// Together these deadlock a common topology: two edge peers (browsers behind
// NAT) connected only through a pinning relay. This test reproduces the
// exact sequence observed live on 2026-07-22 with gossipsub debug logging.
import assert from 'node:assert/strict'
import { rm, mkdtemp } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { createLibp2p } from 'libp2p'
import { tcp } from '@libp2p/tcp'
import { noise } from '@chainsafe/libp2p-noise'
import { yamux } from '@chainsafe/libp2p-yamux'
import { identify } from '@libp2p/identify'
import { gossipsub } from '@libp2p/gossipsub'
import { createHeliaLight } from 'helia'
import { withBitswap } from '@helia/bitswap'
import { withHTTP } from '@helia/http'
import { withLibp2p } from '@helia/libp2p'
import { createOrbitDB } from '@orbitdb/core'
import * as dagCbor from '@ipld/dag-cbor'
import * as dagJson from '@ipld/dag-json'
import * as json from 'multiformats/codecs/json'
import { sha512 } from 'multiformats/hashes/sha2'
const waitFor = async (check, timeoutMs = 15000, intervalMs = 200) => {
const deadline = Date.now() + timeoutMs
for (;;) {
const value = await check()
if (value) return value
if (Date.now() > deadline) throw new Error(`waitFor timed out after ${timeoutMs}ms`)
await new Promise((resolve) => setTimeout(resolve, intervalMs))
}
}
const createNode = async (dir) => {
const libp2p = await createLibp2p({
addresses: { listen: ['/ip4/127.0.0.1/tcp/0'] },
transports: [tcp()],
connectionEncrypters: [noise()],
streamMuxers: [yamux()],
services: {
identify: identify(),
pubsub: gossipsub({ emitSelf: false, allowPublishToZeroTopicPeers: true })
}
})
const ipfs = await withBitswap(
withLibp2p(
withHTTP(createHeliaLight({ codecs: [dagCbor, dagJson, json], hashers: [sha512] })),
libp2p
)
).start()
const orbitdb = await createOrbitDB({ ipfs, directory: join(dir, 'orbitdb') })
return { libp2p, ipfs, orbitdb }
}
const stopNode = async (node) => {
try { await node.orbitdb.stop() } catch {}
try { await node.ipfs.stop() } catch {}
try { await node.libp2p.stop() } catch {}
}
describe('upstream @orbitdb/core: Sync misses pre-existing subscribers', function () {
this.timeout(120000)
let tempRoot
let writerA, readerB, relayR
before(async () => {
tempRoot = await mkdtemp(join(tmpdir(), 'orbitdb-sync-repro-'))
writerA = await createNode(join(tempRoot, 'a'))
readerB = await createNode(join(tempRoot, 'b'))
relayR = await createNode(join(tempRoot, 'r'))
})
after(async () => {
await Promise.allSettled([writerA, readerB, relayR].map(stopNode))
if (tempRoot) await rm(tempRoot, { recursive: true, force: true })
})
it('reader connected only via a relay never receives the first entry', async () => {
const relayAddr = relayR.libp2p.getMultiaddrs()
// The live failure was a timing race — the writer published its first
// entry in the ~200 ms window before the relay (the only interconnection
// point) had (re)subscribed to the topic. We force the same end-state
// deterministically by isolating the writer only for the publish.
// 1. Star topology A—R—B. A creates the DB; R and B open it by address
// (manifest propagates A→R→B). Everyone subscribes; a gossipsub mesh
// settles. All empty.
await writerA.libp2p.dial(relayAddr)
await readerB.libp2p.dial(relayAddr)
const dbA = await writerA.orbitdb.open('sync-repro', { type: 'keyvalue' })
const address = dbA.address.toString()
const dbR = await relayR.orbitdb.open(address)
const dbB = await readerB.orbitdb.open(address)
// R has both A and B in its mesh — the exact state the live logs showed
// (GRAFT: Add mesh link from A / from B), and R has done its one-shot
// heads exchange with each while empty.
await waitFor(() => dbR.peers.size >= 2 && dbB.peers.size >= 1)
const relayMesh = () => relayR.libp2p.services.pubsub.mesh?.get?.(address)
await waitFor(() => (relayMesh()?.size ?? 0) >= 2)
// 2. Isolate the writer, then append the FIRST entry. `sync.add` publishes
// it, but A now has no connected peer — the publish reaches nobody.
await writerA.libp2p.hangUp(relayR.libp2p.peerId)
await waitFor(() => writerA.libp2p.getConnections(relayR.libp2p.peerId).length === 0)
await dbA.put('k', 'first-entry')
// 3. Let A's gossipsub message cache (mcache) expire so that reconnecting
// cannot rescue B via IHAVE/IWANT gossip — this isolates the pure
// heads-exchange path, which is exactly how the relay obtained the
// entry live.
await new Promise((resolve) => setTimeout(resolve, 8000))
// 4. A reconnects. R was never unsubscribed, but A disconnected, so R
// cleared A from its Sync `peers` cache (peer:disconnect) — on
// reconnect the subscription replay fires a subscription-change and R
// performs a fresh heads exchange with A. R now HAS the entry, obtained
// via the exchange protocol, NOT via a gossipsub publish.
await writerA.libp2p.dial(relayAddr)
await waitFor(async () => {
const all = await dbR.all()
return all.length === 1 ? all : null
})
// 5. THE BUG. B is connected to R, subscribed, and in R's gossipsub mesh —
// but R got the entry via the heads exchange and never re-announces it,
// and B was never disconnected, so it is still in R's one-shot `peers`
// cache and is never re-exchanged with. B stays empty.
await new Promise((resolve) => setTimeout(resolve, 10000))
const bEntries = await dbB.all()
assert.equal(
bEntries.length,
0,
`expected the deadlock: B must NOT have the entry, got ${bEntries.length} ` +
'(1 would mean the bug is fixed or gossip rescued it — see the issue analysis)'
)
// 6. Control — the data is one `sync.add` away. Re-announcing the relay's
// heads (the workaround this repo ships) delivers to B within moments,
// proving the deadlock is a Sync coordination gap, not a transport fault.
for (const head of await dbR.log.heads()) {
await dbR.sync.add(head)
}
await waitFor(async () => {
const all = await dbB.all()
return all.length === 1 ? all : null
})
})
})
Sync never delivers the first entry to a reader connected only through a relay
Package:
@orbitdb/core@4.0.0(src/sync.js)Summary
In a star topology where two peers can reach each other only through a third
(relay/pinner) — the common browsers-behind-NAT case — a reader can be
permanently unable to receive the writer's first database entry, even though
both peers are connected to the relay, subscribed to the database topic, and
grafted into the relay's gossipsub mesh. The data sits in the relay's OrbitDB
and is never delivered onward.
The root cause is that
Synccouples "receive an entry" and "announce an entry"too tightly to the pubsub publish path and to the one-shot
subscription-changeedge:sync.add()publishes only locally-appended entries (sync.js:257).Entries obtained via the heads-exchange protocol are never re-announced on
the topic — so a middleman that learns an entry by exchange never forwards
it via pubsub.
sync.js:192,peers.has(peerId) → return). If a peer exchanges heads while the other sideis still empty, it never re-exchanges when heads appear later (only a
peer:disconnectclears the cache,sync.js:246).sync.js:291):startSynconly reactsto
subscription-changeevents after attaching its listener; a peer thatwas already subscribed when the database opens never triggers an exchange.
Individually each is defensible; together they deadlock the topology above.
What happens
Writer A, reader B, relay R. A—R and B—R only (A and B never meet).
instant (or A's mesh/fanout has no subscribed target yet), the single
pubsub.publishreaches nobody. There is no retry and no later re-announce.pubsub). Per (1), R does not publish it onward.
R's one-shot
peerscache and is never re-exchanged with. Per (3), nothingre-triggers it.
B is connected, subscribed and mesh-grafted — and never receives the entry.
Reproduction
Deterministic, local, ~20 s, only
@orbitdb/core+helia+libp2p:Permalink (runnable mocha spec): https://github.com/NiKrause/orbitdb-relay/blob/1f393a1/mocha/upstream-sync-preexisting-subscribers.mjs
The construction forces the end-state without depending on scheduler timing:
open the database on all three nodes and settle a gossipsub mesh (the relay
grafts both peers); isolate the writer and append the first entry so its
publish reaches nobody; let the writer's gossipsub message cache (
mcache)expire so a later reconnect cannot rescue the reader via
IHAVE/IWANT;reconnect the writer so the relay obtains the entry via the heads exchange
(not a publish); assert the reader stays empty (the bug); then a single
dbRelay.sync.add(head)delivers it — proving aSynccoordination gap, not atransport fault.
3/3runs green (green = bug reproduced).Field evidence
Observed on a live relay (browser on a CI runner + browser on a cloud VM,
connected only via a WebSocket/circuit-relay pinning node), gossipsub debug on
the relay, on a brand-new database topic
T:The mesh does form — ~3 s too late for the first publish — and nothing
re-announces the entry afterward.
Suggested fixes (any one breaks the deadlock)
startSync: enumerate already-subscribed peers. Call the samejoin/exchange path for
pubsub.getSubscribers(address)at start, instead ofrelying only on the
subscription-changeedge.(or expose a supported "request re-exchange" that does not require
stop()/start(), which also tears down the subscription).pinner that learns an entry by exchange forwards it via pubsub.
Workaround we ship
In our relay: after a sync pass that received updates, re-announce the current
heads via
db.sync.add(head), so every current subscriber receives themregardless of exchange ordering (receivers de-duplicate via
joinEntry).Note: keeping the relay pubsub-subscribed across database closes does not
work — because discovery is edge-triggered, a standing subscription turns the
next open into a no-op change and no peer ever dials the exchange (we reproduced
this as a hard regression).
Full reproduction test (self-contained mocha spec)