feat(serverless): network-volume warm cache (VolumeCache) by deanq · Pull Request #531 · runpod/runpod-python

deanq · 2026-07-06T23:00:10Z

Summary

Adds VolumeCache, a small explicit primitive that warms local directories (e.g. a model cache) across serverless workers using a mounted network volume. It turns a repeated multi-GB model download on every cold start into a one-time cost per endpoint. Stdlib-only, best-effort: any failure degrades to a cold worker and never raises into your handler.

Interface — a closure

from runpod.serverless import VolumeCache

with VolumeCache(dirs=["/root/.cache/huggingface"]):
    model = load_model()

__enter__ → hydrate(): reconcile volume → container (copy files missing/newer in the container).
__exit__ → sync(background=True): reconcile container → volume on a daemon thread so your code returns immediately; an atexit join (bounded by a timeout) completes it before the process ends.

How it works

The volume holds a browsable mirror of the cached directories at {volume}/.cache/{namespace}/… (namespace defaults to RUNPOD_ENDPOINT_ID). No opaque archives, no accumulation.
Reconcile transfers only the diff (files missing or newer by size/mtime), via per-file atomic copy (copy2 to a unique temp + os.replace). Concurrent workers are last-writer-wins with no torn files.
Safety: skips symlinks; hydrate only writes destinations that resolve under a configured directory; namespace must be a single safe path component.

Notes

No new runtime dependencies; honors the >=3.10 floor.
Explicit, opt-in by design: users add the with block where it fits. There is no automatic worker-loop wiring or environment-variable toggle — an earlier draft had a built-in; it was removed in favor of the single explicit primitive. Auto-wiring can be layered on later if desired.

Test plan

Unit suite tests/test_serverless/test_utils/test_rp_volume_cache.py: availability gating, namespace validation, sync→hydrate round-trip, reconcile idempotence (unchanged files not recopied), newer-file protection, path exclusions, symlink skip, hydrate destination-containment (escape attempt blocked), context-manager enter/exit (+ exception propagation), background completion via the atexit join (bounded), temp-file exclusion, and best-effort swallow vs re-raise.
Full tests/test_serverless suite green; repo make quality-check passes (557 tests, ~94% overall, ~98% on the new module).
Tests are stdlib + tmp_path only — no Docker, no network.

Docs

docs/serverless/volume_cache.md and a short README section under Serverless.

…meCache delta

… marker

…floor

…0 floor

…mlinks

…ker loop

…test state Wrap build_default_cache() parse+construct in try/except to prevent malformed RUNPOD_VOLUME_CACHE_MAX_GB from crashing worker startup; degrades to cache-disabled instead. Add sync() no-op to fake cache in test_run_worker_hydrates_registered_cache and wrap test body with reset_builtin_state_for_test() to prevent module-state leakage to subsequent tests (sync_after_job could fire and hit AttributeError).

…all outputs

capy-ai · 2026-07-06T23:00:19Z

Capy auto-review is paused for this organization because the usage-cycle auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews.

promptless · 2026-07-06T23:10:32Z

Promptless prepared a documentation update related to this change.

Triggered by runpod-python PR #531

Documents the new Serverless network-volume warm cache: a new page covering the VolumeCache API and its warm()/hydrate()/sync() lifecycle, the RUNPOD_VOLUME_CACHE / RUNPOD_CACHE_DIRS / RUNPOD_VOLUME_CACHE_MAX_GB environment variables, and auto-discovered cache directories (HF_HOME / HF_HUB_CACHE / TORCH_HOME), plus a cold-start optimization cross-link. Since this PR is still open, the draft is phrased around the env-var toggle rather than asserting default-on behavior — worth confirming the default-on vs opt-in decision and target SDK version before merge.

Review: Document Serverless network-volume warm cache (VolumeCache)

…che-for-serverless-volumecache

Copilot

Pull request overview

This PR introduces a reusable serverless SDK primitive (VolumeCache) for warming local model/cache directories from a mounted network volume and syncing back deltas as per-worker tar “shards”, then wires it into the serverless worker lifecycle to persist model weights across worker recycling (SLS-367).

Changes:

Added runpod.serverless.utils.rp_volume_cache.VolumeCache with hydrate(), sync(), and warm() plus built-in helpers (build_default_cache, sync_after_job).
Auto-integrated the cache into the worker loop (run_worker hydrates/registers) and job execution (run_job dispatches a one-time async sync after a successful handler run).
Expanded public exports (runpod.serverless.VolumeCache, runpod.serverless.utils.VolumeCache) and added a comprehensive unit test suite.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_serverless/test_utils/test_rp_volume_cache.py	New unit test suite covering VolumeCache behavior, safety, retention, and worker/job wiring.
tests/test_serverless/test_utils_init.py	Updates `runpod.serverless.utils.__all__` expectations to include `VolumeCache`.
tests/test_serverless/test_init.py	Updates `runpod.serverless.__all__` expectations to include `VolumeCache`.
runpod/serverless/worker.py	Hydrates and registers the default cache before starting job scaling.
runpod/serverless/utils/rp_volume_cache.py	New implementation of `VolumeCache` plus built-in env-driven wiring helpers.
runpod/serverless/utils/init.py	Re-exports `VolumeCache` from `runpod.serverless.utils`.
runpod/serverless/modules/rp_job.py	Dispatches a one-time background cache sync after non-streaming jobs.
runpod/serverless/init.py	Re-exports `VolumeCache` from `runpod.serverless`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…e, namespace validation, lint

deanq · 2026-07-07T00:44:32Z

Addressed review feedback in ede2d5c:

Copilot (behavioral):

run_job_generator now dispatches the one-time sync_after_job() on its success path — streaming/generator handlers previously never warmed the cache.
_enforce_retention now tolerates a shard vanishing under concurrent prune (getsize no longer raises FileNotFoundError out of best-effort sync()).
VolumeCache now rejects an unsafe namespace (absolute / path separators / ./..) to prevent shard/marker paths escaping the volume.

CodeQL / code-quality:

Documented the best-effort empty except in the temp-shard sweep; removed the unreachable-code pattern in the warm-on-exception test; consolidated mixed import styles in the test module.
The "unused global _SYNCED" alert is a false positive (_SYNCED is read/written in sync_after_job and reset in reset_builtin_state_for_test) — dismissed.

Tests added for streaming sync, retention resilience, and namespace validation. Full suite green (566 tests, 94% coverage).

Still open for a maintainer decision before merge: default-on vs opt-in for the serverless built-in when a volume is mounted (currently on, disabled via RUNPOD_VOLUME_CACHE=0).

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

…che-for-serverless-volumecache

…var built-in

…emp files

…che-for-serverless-volumecache

… open() in assert, unused global)

deanq added 14 commits July 6, 2026 01:54

feat(serverless): add VolumeCache skeleton with availability gating

fec7fc3

feat(serverless): VolumeCache.sync writes collision-free delta shards

2853f38

fix(serverless): tolerate coarse network-fs mtime granularity in Volu…

f0a76f8

…meCache delta

feat(serverless): VolumeCache.hydrate extracts shards with idempotent…

4b0f6df

… marker

fix(serverless): drop 3.12-only tarfile filter kwarg to honor py3.10 …

b95e1f1

…floor

fix(serverless): apply tar data filter when available, honoring py3.1…

6fc8250

…0 floor

feat(serverless): harden VolumeCache extract against traversal and sy…

5b76d8d

…mlinks

fix(serverless): realpath-normalize cache dirs so symlinked mounts match

b52bb48

feat(serverless): add size-capped retention to VolumeCache

fd19f65

feat(serverless): add best-effort guard and warm() context manager

68b56d3

feat(serverless): export VolumeCache from runpod.serverless

c3a9e98

feat(serverless): auto-hydrate model cache from network volume in wor…

a8f466a

…ker loop

fix(serverless): guard cache sync dispatch, sweep temp shards, cover …

f8fe045

…all outputs

github-advanced-security AI found potential problems Jul 6, 2026

View reviewed changes

github-code-quality Bot found potential problems Jul 6, 2026

View reviewed changes

deanq requested a review from Copilot July 6, 2026 23:48

Merge branch 'main' into deanquinanola/sls-367-network-volume-warm-ca…

53bb5a6

…che-for-serverless-volumecache

Copilot started reviewing on behalf of deanq July 6, 2026 23:48 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

Comment thread runpod/serverless/utils/rp_volume_cache.py

Comment thread runpod/serverless/utils/rp_volume_cache.py Outdated

Comment thread runpod/serverless/modules/rp_job.py Outdated

promptless Bot mentioned this pull request Jul 7, 2026

docs: Document Serverless network-volume warm cache (VolumeCache) runpod/docs#693

Draft

fix(serverless): address review - streaming sync, retention resilienc…

ede2d5c

…e, namespace validation, lint

deanq requested a review from Copilot July 7, 2026 01:39

Copilot started reviewing on behalf of deanq July 7, 2026 01:40 View session

feat(serverless): make network-volume warm cache opt-in; add usage docs

eeb26e6

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread runpod/serverless/utils/rp_volume_cache.py Outdated

Comment thread runpod/serverless/utils/rp_volume_cache.py Outdated

Merge branch 'main' into deanquinanola/sls-367-network-volume-warm-ca…

1e21876

…che-for-serverless-volumecache

deanq added 2 commits July 7, 2026 15:45

refactor(serverless): VolumeCache mirror+reconcile closure; drop env-…

b184d21

…var built-in

fix(serverless): bound atexit sync join, unique temp names, exclude t…

87438a6

…emp files

github-advanced-security AI found potential problems Jul 7, 2026

View reviewed changes

github-code-quality Bot found potential problems Jul 7, 2026

View reviewed changes

deanq and others added 3 commits July 7, 2026 18:10

Merge branch 'main' into deanquinanola/sls-367-network-volume-warm-ca…

ca96ab5

…che-for-serverless-volumecache

fix(serverless): clear CodeQL alerts (empty-except, unreachable test,…

9469d3a

… open() in assert, unused global)

docs(serverless): add VolumeCache warm-cache handler example

351f35d

Uh oh!

Conversation

deanq commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Interface — a closure

How it works

Notes

Test plan

Docs

Uh oh!

capy-ai Bot commented Jul 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

promptless Bot commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deanq commented Jul 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

deanq commented Jul 6, 2026 •

edited

Loading