Skip to content

fix(rate-limit): stop the test-context abort racing the decision (#6236) - #6405

Merged
koala73 merged 1 commit into
koala73:mainfrom
Yigtwxx:fix/rate-limit-abort-margin-test-flake
Aug 10, 2026
Merged

fix(rate-limit): stop the test-context abort racing the decision (#6236)#6405
koala73 merged 1 commit into
koala73:mainfrom
Yigtwxx:fix/rate-limit-abort-margin-test-flake

Conversation

@Yigtwxx

@Yigtwxx Yigtwxx commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the CI flake that turned unit red on #6044 (job): 22600 pass, 1 fail, and the one failure was tests/rate-limit.test.mts — a file that PR does not touch.

✖ paid-provider endpoint policies abort a stalled Redis fetch before failing closed (#6236)
  AssertionError: the Redis transport must be cancelled, not left pending
  false !== true

The endpoint limiter arms two deadlines and their order is load-bearing:

const ENDPOINT_RATE_LIMIT_TIMEOUT_MS  = process.env.NODE_TEST_CONTEXT ? 25 : 5_000;  // :30
const ENDPOINT_REDIS_ABORT_TIMEOUT_MS = process.env.NODE_TEST_CONTEXT ? 20 : 4_500;  // :34

The abort has to fire before the SDK's availability-first race resolves, or the decision returns while the Upstash fetch is still pending in the isolate — which is exactly what the assertion checks.

The two timers do not start at the same moment. timeout: ENDPOINT_RATE_LIMIT_TIMEOUT_MS (:565) starts when rl.limit() is called, but signal: () => AbortSignal.timeout(ENDPOINT_REDIS_ABORT_TIMEOUT_MS) (:560) is a per-request factory the Upstash client only invokes when it builds the request. So the real ordering is armingCost + abortMs against decisionMs, and the configured gap has to cover the arming cost.

In production it does, comfortably: the gap is 500 ms and arming is a few milliseconds. The test-context pair mirrored the production ratio instead of the production margin, leaving 5 ms — and under the node test runner arming costs far more than that, because each case re-imports the module through tsx with a cache-busting query.

I instrumented the exact path with the test's own stub fetch, 10 runs on an idle machine:

before after
fetch (and its abort signal) armed at 8.3 – 71.1 ms 9.5 – 65.8 ms
headroom before the decision deadline 1.1 – 16.3 ms 164.2 – 220.5 ms

1.1 ms of headroom is not a margin, it is a coin flip, and --test-concurrency=16 on a 2-core runner is what flips it.

The fix, and the alternative I rejected

Widen the test-context decision deadline to 250, leaving the abort at 20. The gap becomes 230 ms — larger than the worst arming cost observed — and the comment now says why the test pair deliberately is not the production ratio scaled down.

The alternative was to leave the constants alone and make the assertion poll for the abort with a bounded deadline. I rejected it because it changes what the test claims: the name is "abort ... before failing closed", and an eventual-cancellation assertion would pass even if the ordering regressed for real. Fixing the margin keeps the original claim and makes it deterministic.

Scaling the production ratio down (e.g. 250 / 225) would also be wrong here: it restores a 25 ms gap, which is still inside the observed arming range.

Verification

The new guard is mutation-proven — reverting 250 to 25 with everything else intact:

✖ keeps the Redis abort deadline far enough ahead of the limiter decision (#6236)
  the abort must fire well before the decision resolves; gap is 5ms

Suites:

tests/rate-limit.test.mts                     45 pass, 0 fail  (44 before; ×3 runs)
+ api-key-rate-limit, pro-checkout-rate-limit,
  user-prefs-rate-limit, cors-fail-closed,
  api-plan-limit-readiness, premium-fetch     144 pass, 0 fail

Cost of the wider deadline, measured both ways on the same machine:

tests/rate-limit.test.mts alone      2.50-2.56 s  ->  2.62-2.67 s
the seven files above, concurrent    2.79 s       ->  2.95-3.02 s
per file, run individually           no difference outside noise

Only the one case that genuinely waits out the decision deadline (fail closed when Redis ignores abort until the SDK timeout) pays for it; every other path still resolves on the 20 ms abort.

Other gates:

npm run typecheck        clean
npx biome check          clean (2 files)
npm run lint:boundaries  no violations
npm run lint:unicode     2791 files scanned, clean

npm run lint:rate-limit-policies is not runnable on Windows locally — scripts/enforce-rate-limit-policies.mjs builds a file:///C:/C:/... URL and fails with ERR_MODULE_NOT_FOUND on a clean checkout of main too, before this change. It is unaffected either way: no policy table is touched.

Out of scope

  • The production values. 5_000 / 4_500 are unchanged. The bug is in the test-context pair only.
  • The lazy signal factory. Arming the abort at limit() time instead of at request-build time would remove the dependency on arming cost entirely, but it means reaching past the Upstash client's own request lifecycle. The margin fix is the smaller change and it closes the observed failure.
  • The global limiter (getRatelimit) and checkScopedRateLimit, which do not use these deadlines.
  • Every other test in the file, and the api/_rate-limit.js mirror, which only references these constants in a comment.

Type of change

  • Bug fix
  • New feature
  • New data source / feed
  • New map layer
  • Refactor / code cleanup
  • Documentation
  • CI / Build / Infrastructure

Affected areas

  • Map / Globe
  • News panels / RSS feeds
  • AI Insights / World Brief
  • Market Radar / Crypto
  • Desktop app (Tauri)
  • API endpoints (/api/*) — the endpoint rate limiter's test-context deadlines only; production behaviour is byte-identical
  • Config / Settings
  • Other: CI reliability — unit on the full suite

Checklist

  • Tested on worldmonitor.app variant — N/A. The change is inert outside NODE_TEST_CONTEXT; there is no runtime or user-visible surface to exercise.
  • Tested on tech.worldmonitor.app variant (if applicable) — N/A, no variant-specific behaviour.
  • New RSS feed domains added to api/rss-proxy.js allowlist (if adding feeds) — N/A, no feeds added.
  • No API keys or secrets committed
  • TypeScript compiles without errors (npm run typecheck)

Documentation Alignment Checklist

N/A — no published documentation claim changes. Nothing here is documented, published, or read by a consumer: the values are internal test-context constants and the new export is test-only. Listed for completeness:

  • Claim ledger attached or linked — N/A, no documented claim changes.
  • All required Audit Council role signoffs attached — N/A, no methodology or contract change.
  • Generated docs regenerated from proto where applicable — N/A, no proto change.
  • Fixture-backed examples recomputed — N/A, no published example depends on these deadlines.
  • Redis writers/readers enumerated for every documented key — N/A, no key is added, removed, or written differently. The change alters only how long the limiter waits before abandoning a Redis call under the test runner.

…la73#6236)

The endpoint limiter arms two deadlines: the SDK's availability-first
decision race (ENDPOINT_RATE_LIMIT_TIMEOUT_MS) and an AbortSignal.timeout
that cancels the underlying Upstash fetch just before it
(ENDPOINT_REDIS_ABORT_TIMEOUT_MS). The abort must land first or the
transport is left pending in the isolate, which is what
tests/rate-limit.test.mts pins.

The ordering held in production but not under the node test runner.
Production's 500ms gap dwarfs the few milliseconds the Upstash client
needs to build a request and call the signal factory, and the abort timer
does not start until that call while the decision timer starts at limit().
The NODE_TEST_CONTEXT pair mirrored the production ratio at 25/20, which
left a 5ms gap against an arming cost measured at 8-71ms, so as little as
1.1ms of headroom survived. Under --test-concurrency=16 that inverted and
the suite went red on an assertion about the transport, not about the
limiter.

Widen the test-context decision deadline to 250ms so the gap exceeds the
arming cost rather than mirroring a ratio that only works when arming is
free. The abort stays at 20ms, so failure paths still resolve fast: the
one test that waits out the full decision deadline costs ~120ms more and
nothing else changes.
@vercel

vercel Bot commented Aug 10, 2026

Copy link
Copy Markdown

@Yigtwxx is attempting to deploy a commit to the World Monitor Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added the trust:safe Brin: contributor trust score safe label Aug 10, 2026
@koala73
koala73 merged commit 87a2372 into koala73:main Aug 10, 2026
29 of 30 checks passed
@koala73

koala73 commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Thank you @Yigtwxx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

trust:safe Brin: contributor trust score safe

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants