fix(cross-source-signals): unwrap seed envelopes and align extractors with their writers (#5870) - #5896

Summary

Fixes #5870.

readAllSourceKeys() bare-parsed every value it read:

try { data[SOURCE_KEYS[i]] = JSON.parse(raw); } catch { /* skip malformed */ }

Most of those keys are written by contract-mode seeders, which store { _seed, data } (scripts/_seed-utils.mjs:499). So d['<key>'] was the envelope, payload.clusters / payload.quotes / payload.threats were undefined, the Array.isArray(...) guard on the next line was false, and the extractor returned []. No throw, so the try/catch in the aggregator never logged. The seeder still exited 0 and published a shorter array.

Auditing all of it turned up a second, independent layer: even after a correct unwrap, most extractors read field names and enum spellings their writer has never published. extractWildfireEscalation is the clearest case — it filters on f.radiativePower > 5000 || f.severity === 'extreme' against a payload whose detections carry frp and have no severity field at all, with FRP reported in MW (the writer's own significance threshold is 80, not 5000). Only its brightness > 400 clause was ever correct, which is why the code reads plausibly and produces nothing.

So this is three fixes on one seeder: unwrap the envelope, align every extractor with its writer, and make the suite able to see both.

The audit

The issue asks for all source keys to be checked against their writers, so here is the result. "Enveloped" = the writer passes declareRecords to runSeed, which is what turns on contract mode (_seed-utils.mjs:1758).

military:flights:stale:v1 is now registered in SOURCE_KEYS. extractMilitaryFlightSurge already falls back to it, but it was never fetched, so d['military:flights:stale:v1'] was always undefined. The writer publishes the same payload to both keys (seed-military-flights.mjs:1327) with LIVE_TTL = 600 against this seeder's 15-minute cadence, so the live key is absent on most runs and the stale key is the one that carries the data. seed-correlation.mjs:14 registers the same pair.

intelligence:gdelt-intel:v1 is dropped from SOURCE_KEYS: its only reader was the media-tone fallback, which read topic.avgTone || topic.tone against topic objects shaped { id, articles, fetchedAt, attemptedAt, _tone, _vol } (seed-gdelt-intel.mjs:438). safeNum() returned 0, 0 > -3 skipped every topic, so the branch could not fire even after the unwrap. Rebuilding it on topic._tone would mean re-deriving the trend and staleness rules the per-topic path already implements under #5478/#5863 review, so it is removed rather than half-restored.

Why JSON.parse stays outside unwrapEnvelope

unwrapEnvelope accepts a raw string, so unwrapEnvelope(raw).data would have been shorter. It also returns that string as data when the parse fails, which would register a malformed value as a found key and inflate the Found N/M source keys populated line. Parsing first preserves the existing skip-malformed behaviour exactly, and unwrapEnvelope short-circuits its own string branch when handed an object, so nothing is parsed twice.

Legacy safety is structural rather than by-key: unwrapEnvelope only unwraps when _seed.fetchedAt is a number (_seed-envelope-source.mjs:59). The one live payload that looks like a half-envelope is gdelt:intel:tone:* — a top-level data array plus a fetchedAt, no _seed — and there is a dedicated test pinning that it passes through untouched, since unwrapping it would kill the one tone path that works today.

Design decisions for maintainer review

These are calibration questions, not read fixes. I took the conservative option in each case and left the alternative here rather than deciding it in code.

1. Displacement is left silent. displacement:summary:v1:<year> publishes an annual UNHCR stock — summary.countries[] with refugees/idps/totalDisplaced (seed-displacement-summary.mjs:175) — and carries no flow, delta or trend field anywhere. The extractor read crises[].newDisplacements and .trend, which the key has never published. This is not a rename away from working: a surge is a flow, and picking a totalDisplaced cutoff to stand in for one would invent a calibration rather than repair a read. It returns [] with the reason in place. The alternative is to redefine the signal as stock-level and rename it; happy to do that here or in a follow-up if you prefer.
2. Wildfire threshold. Rather than rescale frp > 5000, this uses possibleExplosion — the significance flag the writer itself publishes (frp > 80 && brightness > 380). The per-theater count < 5 floor and the /50 score divisor are untouched, but both were calibrated against a shape that never produced anything, so they have never actually been exercised.
3. Shipping. Same principle: the payload has no route or disruption concept, so this keys off the writer's own spikeAlert rather than a new changePct threshold.
4. Radiation fires on SPIKE only, not ELEVATED. BASE_WEIGHT is 3.5, so a single ELEVATED reading would score CRITICAL on its own.
5. Weather theater is North America. NWS alerts carry areaDesc ("Kern County, CA; Tulare County, CA") and no country or region. Bucketing on that string gives almost every alert its own theater, which can never join a composite; the feed is US-only, so the theater is fixed by construction.
6. Military flight theater uses operatorCountry. That is whose air force is flying, not where — the closest proxy this payload exposes. The writer also publishes geo-bucketed theaters to theater-posture:sebuf:v1, which would be more accurate and a larger change; say the word and I will use it instead.
7. Forecast deterioration keeps probability as the only criterion. computeTrends (seed-forecasts.mjs:2720) only assigns stable/rising/falling, so trend === 'deteriorating' and direction === 'negative' were dead. Whether rising should also qualify is your call.

Enum comparisons use dual-accept (OUTAGE_SEVERITY_MAJOR and major both match), following seed-correlation.mjs:212. That makes every change strictly additive — no signal that fires today can stop firing.

Why the tests could not see this

Both existing suites reconstructed the module with readFileSync + regex + vm.runInContext, because runSeed(...) ran unconditionally at import time. One of those regexes deleted readAllSourceKeys outright:

.replace(/async function readAllSourceKeys[\s\S]*?\r?\n}\r?\n\r?\n\/\/ ── Signal extractors/m, '// readAllSourceKeys removed for unit test\n\n// ── Signal extractors')

and the fixtures were hand-built bare payloads, i.e. the pre-envelope world, at exactly the seam where the defect lives.

So this adds the direct-run guard 118 of the 163 scripts/seed-*.mjs files already use (seed-regulatory-actions.mjs:319 is the template) and exports the extractors, and both suites now import the real module. The guard is a no-op in production: scripts/_bundle-runner.mjs spawns each section as its own process with the script path as argv[1], and seed-china-decision-signals.mjs:155 in this same bundle is both guarded and imported by seed-bundle-derived-signals.mjs:3. Existing assertions are unchanged, except that the extractor-registration check now asserts membership in the exported EXTRACTORS registry instead of grepping the source text.

Verification

Targeted suites, all on this branch:

tests/cross-source-signals-envelope-reads.test.mjs    46 pass, 0 fail
tests/cross-source-signals-wildfire.test.mjs           6 pass, 0 fail
tests/cross-source-signals-regulatory.test.mjs         7 pass, 0 fail
tests/cross-source-signals-tone-staleness.test.mjs     5 pass, 0 fail
                                                      64 pass, 0 fail

Every guard is mutation-proven. Reverting any one of the 23 corrections to the code it replaced turns the suite red — 23 mutants, no survivors:

Three of those only started failing after the fixture matrix asserted the expected theater and payload-sourced detectedAt rather than just non-empty output — military, earthquake and gpsjam all survived a "does it fire" assertion, because a signal that silently falls back to theater: 'Global' and detectedAt: Date.now() still fires. That is precisely how the military-flight signal sat outside every theater composite, so the assertions are pinned exactly.

Every fixture is field-for-field the record its writer builds, with a file:line provenance comment, and each one is driven through the exact transform readAllSourceKeys performs rather than hand-unwrapped. Enveloped fixtures additionally assert that the raw envelope produces nothing, which pins the bug itself.

Other gates:

npm run typecheck        clean
npx biome check          clean (5 files)
npm run lint:boundaries  no violations
check-unicode-safety     2601 files scanned, clean

npm run test:data: identical failure set to origin/main — 45 failing test names on both, comm diff empty in both directions (OpenAPI contract, docs/i18n, pricing and Docker suites that are already red on a clean checkout).

The seeder was not run against production Redis.

Out of scope

• scripts/seed-correlation.mjs:40 reads its own INPUT_KEYS with the same bare JSON.parse and sits next to this file in the same Railway bundle. Its inputs are mostly legacy bare keys so the blast radius is smaller, but it is the same defect and worth its own issue.
• The MAX_SIGNALS = 30 cap and the per-extractor .slice(0, 2..5) limits are untouched. More signals will now reach list-cross-source-signals and the composite detector — that is the point of the fix — but both consumers already take a bounded array, and changing the caps in the same PR would mix a bug fix with a tuning change.
• Extractor thresholds beyond the seven listed above are left exactly as they were.
• The _country-brief-context / prompt-context work in fix(intel): newline forges prompt rows in three sibling prompt-context modules (out of #5857's scope) #5881 is unrelated and not touched here.

Type of change

• Bug fix
• New feature
• New data source / feed
• New map layer
• Refactor / code cleanup
• Documentation
• CI / Build / Infrastructure

Affected areas

• Map / Globe
• News panels / RSS feeds
• AI Insights / World Brief
• Market Radar / Crypto
• Desktop app (Tauri)
• API endpoints (/api/*)
• Config / Settings
• Other: scripts/seed-cross-source-signals.mjs (Railway seeder feeding intelligence:cross-source-signals:v1)

Checklist

• Tested on worldmonitor.app variant — N/A. This is a Railway seeder; its output is only observable after a scheduled run writes intelligence:cross-source-signals:v1. Verified through the extractor suites instead, against fixtures derived field-for-field from each writer.
• Tested on tech.worldmonitor.app variant (if applicable) — N/A, no variant-specific behaviour.
• New RSS feed domains added to api/rss-proxy.js allowlist (if adding feeds) — N/A, no feeds added.
• No API keys or secrets committed
• TypeScript compiles without errors (npm run typecheck)

Documentation Alignment Checklist

N/A — this PR does not publish or change any documentation claim. It changes how one seeder reads existing Redis keys and does not alter the published shape of intelligence:cross-source-signals:v1, the OpenAPI contract for list-cross-source-signals, methodology, or any generated doc. Listed for completeness:

• Claim ledger attached or linked — N/A, no documented claim changes.
• All required Audit Council role signoffs attached — N/A, no methodology or contract change.
• Generated docs regenerated from proto where applicable — N/A, no proto change.
• Fixture-backed examples recomputed — N/A, no published example depends on this seeder's input reads.
• Redis writers/readers enumerated for every documented key — done in the audit table above: every SOURCE_KEYS entry is mapped to its writer and its envelope mode.