Skip to content

feat(caching): new serialization format#20970

Open
avivkeller wants to merge 1 commit into
mainfrom
test-different-caching-system
Open

feat(caching): new serialization format#20970
avivkeller wants to merge 1 commit into
mainfrom
test-different-caching-system

Conversation

@avivkeller

Copy link
Copy Markdown
Member

Tip

I'm testing how this PR impacts benchmarks. Local benchmarks show a 4-5x improvement in caching time, but I want to open a PR to validate those gains.

Replaces webpack's persistent cache serialization stack and filesystem cache layout with a new compact binary serializer and segmented disk cache format.

The old middleware/object-serializer based pack-file cache is replaced by:

  • DiskCacheStrategy, which stores a validated cache index plus immutable cache segments.
  • lib/serialization/{Encoder,Decoder,Reader,Writer,FileStore,Lazy,TypeRegistry}, which provide the new binary serialization system.
  • lib/cache/format/{CacheIndex,CacheSegment,SegmentManager}, which track entries, segment metadata, build snapshots, dependency resolution state, garbage collection, and compaction.
  • updated serializable classes that now use the new serialize/deserialize context API.
  • focused unit coverage for built-in values, object references, circular references, registered serializers, lazy values, separate file-store values, and non-serializable values.

@changeset-bot

changeset-bot Bot commented May 16, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 26efab0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
webpack Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

Copy link
Copy Markdown
Contributor

This PR is packaged and the instant preview is available (26efab0).

Install it locally:

  • npm
npm i -D webpack@https://pkg.pr.new/webpack@26efab0
  • yarn
yarn add -D webpack@https://pkg.pr.new/webpack@26efab0
  • pnpm
pnpm add -D webpack@https://pkg.pr.new/webpack@26efab0

@github-actions

Copy link
Copy Markdown
Contributor

Types Coverage

Coverage after merging test-different-caching-system into main will be
98.81%
Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
bin
   webpack.js98.77%100%100%98.77%91
examples
   build-common.js100%100%100%100%
   buildAll.js100%100%100%100%
   examples.js100%100%100%100%
   template-common.js98.21%100%100%98.21%72
examples/custom-javascript-parser
   test.filter.js100%100%100%100%
examples/custom-javascript-parser/internals
   acorn-parse.js100%100%100%100%
   meriyah-parse.js100%100%100%100%
   oxc-parse.js91.30%100%100%91.30%140, 142–143, 145, 147, 153–154, 161, 168, 90
examples/markdown
   webpack.config.mjs100%100%100%100%
examples/typescript
   test.filter.js50%100%100%50%5
examples/virtual-modules
   test.filter.js100%100%100%100%
examples/wasm-bindgen-esm
   test.filter.js100%100%100%100%
examples/wasm-complex
   test.filter.js100%100%100%100%
examples/wasm-simple
   test.filter.js100%100%100%100%
examples/wasm-simple-source-phase
   test.filter.js100%100%100%100%
lib
   APIPlugin.js100%100%100%100%
   AsyncDependenciesBlock.js100%100%100%100%
   AutomaticPrefetchPlugin.js100%100%100%100%
   BannerPlugin.js100%100%100%100%
   Cache.js98.21%100%100%98.21%101
   CacheFacade.js100%100%100%100%
   Chunk.js99.72%100%100%99.72%37
   ChunkGraph.js100%100%100%100%
   ChunkGroup.js100%100%100%100%
   ChunkTemplate.js100%100%100%100%
   CleanPlugin.js98.72%100%100%98.72%206, 226, 382
   CodeGenerationResults.js100%100%100%100%
   CompatibilityPlugin.js100%100%100%100%
   Compilation.js98.55%100%100%98.55%1554, 1850, 1857, 1865, 1887, 2783, 3208, 3870, 3899, 3952–3953, 3957, 3962, 3978–3979, 3993–3994, 3999–4000, 4477, 4503, 493, 498, 5211, 5292, 5307, 5332–5333, 5335, 5659, 5664, 5670, 5673, 5685, 5687, 5691, 5707, 5722, 5754, 5808, 5832, 5946, 712–713
   Compiler.js99.55%100%100%99.55%1116–1117, 1125
   ConcatenationScope.js98.59%100%100%98.59%189
   ConditionalInitFragment.js100%100%100%100%
   ConstPlugin.js100%100%100%100%
   ContextExclusionPlugin.js100%100%100%100%
   ContextModule.js100%100%100%100%
   ContextModuleFactory.js97.75%100%100%97.75%258, 393, 418, 443, 447, 458
   ContextReplacementPlugin.js100%100%100%100%
   DefinePlugin.js98.92%100%100%98.92%158–159, 175, 194, 268
   DependenciesBlock.js100%100%100%100%
   Dependency.js98.20%100%100%98.20%379, 425
   DependencyTemplate.js100%100%100%100%
   DependencyTemplates.js100%100%100%100%
   DotenvPlugin.js97.88%100%100%97.88%237, 378, 391–392
   DynamicEntryPlugin.js100%100%100%100%
   EntryOptionPlugin.js100%100%100%100%
   EntryPlugin.js100%100%100%100%
   Entrypoint.js100%100%100%100%
   EnvironmentPlugin.js97.14%100%100%97.14%49
   ErrorHelpers.js100%100%100%100%
   EvalDevToolModulePlugin.js100%100%100%100%
   EvalSourceMapDevToolPlugin.js100%100%100%100%
   ExportsInfo.js100%100%100%100%
   ExportsInfoApiPlugin.js100%100%100%100%
   ExternalModule.js99.65%100%100%99.65%419, 585
   ExternalModuleFactoryPlugin.js100%100%100%100%
   ExternalsPlugin.js100%100%100%100%
   FileSystemInfo.js99.50%100%100%99.50%182, 2252–2253, 2256, 2267, 2278, 2289, 278, 3694, 3709, 3733
   FlagAllModulesAsUsedPlugin.js100%100%100%100%
   FlagDependencyExportsPlugin.js98.74%100%100%98.74%399, 401, 405
   FlagDependencyUsagePlugin.js100%100%100%100%
   FlagEntryExportAsUsedPlugin.js100%100%100%100%
   Generator.js100%100%100%100%
   HotModuleReplacementPlugin.js100%100%100%100%
   HotUpdateChunk.js100%100%100%100%
   IgnorePlugin.js100%100%100%100%
   IgnoreWarningsPlugin.js100%100%100%100%
   InitFragment.js100%100%100%100%
   JavascriptMetaInfoPlugin.js100%100%100%100%
   LibraryTemplatePlugin.js100%100%100%100%
   LoaderOptionsPlugin.js100%100%100%100%
   LoaderTargetPlugin.js100%100%100%100%
   MainTemplate.js100%100%100%100%
   ManifestPlugin.js100%100%100%100%
   Module.js98.50%100%100%98.50%1304, 1309, 1370, 1384, 1446, 1455
   ModuleFactory.js100%100%100%100%
   ModuleFilenameHelpers.js98.85%100%100%98.85%106, 108
   ModuleGraph.js99.73%100%100%99.73%1004
   ModuleGraphConnection.js100%100%100%100%
   ModuleInfoHeaderPlugin.js100%100%100%100%
   ModuleProfile.js100%100%100%100%
   ModuleSourceTypeConstants.js100%100%100%100%
   ModuleTemplate.js100%100%100%100%
   ModuleTypeConstants.js100%100%100%100%
   MultiCompiler.js99.69%100%100%99.69%645
   MultiStats.js100%100%100%100%
   MultiWatching.js100%100%100%100%
   NoEmitOnErrorsPlugin.js100%100%100%100%
   NodeStuffPlugin.js100%100%100%100%
   NormalModule.js97.78%100%100%97.78%1020, 1036, 1123, 1774, 1779–1789, 708, 711, 728, 745, 986
   NormalModuleFactory.js99.47%100%100%99.47%1074, 1383, 473, 485
   NormalModuleReplacementPlugin.js100%100%100%100%
   NullFactory.js100%100%100%100%
   OptimizationStages.js100%100%100%100%
   OptionsApply.js100%100%100%100%
   Parser.js100%100%100%100%
   PlatformPlugin.js100%100%100%100%
   PrefetchPlugin.js100%100%100%100%
   ProgressPlugin.js98.85%100%100%98.85%519–520, 525, 527, 591
   ProvidePlugin.js100%100%100%100%
   RawModule.js100%100%100%100%
   RecordIdsPlugin.js100%100%100%100%
   RequestShortener.js100%100%100%100%
   ResolverFactory.js100%100%100%100%
   RuntimeGlobals.js100%100%100%100%
   RuntimeModule.js100%100%100%100%
   RuntimePlugin.js100%100%100%100%
   RuntimeTemplate.js100%100%100%100%
   SelfModuleFactory.js100%100%100%100%
   SingleEntryPlugin.js100%100%100%100%
   SourceMapDevToolModuleOptionsPlugin.js100%100%100%100%
   SourceMapDevToolPlugin.js99.16%100%100%99.16%267–268, 610
   Stats.js100%100%100%100%
   Template.js100%100%100%100%
   TemplatedPathPlugin.js98.86%100%100%98.86%134–135
   UseStrictPlugin.js100%100%100%100%
   WarnCaseSensitiveModulesPlugin.js100%100%100%100%
   WarnDeprecatedOptionPlugin.js100%100%100%100%
   WarnNoModeSetPlugin.js100%100%100%100%
   WatchIgnorePlugin.js100%100%100%100%
   Watching.js100%100%100%100%
   WebpackError.js100%100%100%100%
   WebpackIsIncludedPlugin.js100%100%100%100%
   WebpackOptionsApply.js100%100%100%100%
   WebpackOptionsDefaulter.js100%100%100%100%
   buildChunkGraph.js99.87%100%100%99.87%325
   cli.js98.71%100%100%98.71%117, 469, 501, 543, 813
   index.js99.72%100%100%99.72%670
   validateSchema.js94.67%100%100%94.67%100, 87, 89, 98
   webpack.js97.22%100%100%97.22%196, 218, 220
lib/asset
   AssetBytesGenerator.js100%100%100%100%
   AssetBytesParser.js100%100%100%100%
   AssetGenerator.js100%100%100%100%
   AssetModulesPlugin.js97.77%100%100%97.77%285, 309, 312, 364, 40
   AssetParser.js100%100%100%100%
   AssetSourceGenerator.js100%100%100%100%
   AssetSourceParser.js100%100%100%100%
   RawDataUrlModule.js100%100%100%100%
lib/async-modules
   AsyncModuleHelpers.js100%100%100%100%
   AwaitDependenciesInitFragment.js100%100%100%100%
   InferAsyncModulesPlugin.js100%100%100%100%
lib/cache
   AddBuildDependenciesPlugin.js100%100%100%100%
   AddManagedPathsPlugin.js100%100%100%100%
   DiskCacheStrategy.js97.73%100%100%97.73%139, 145, 155, 271, 275, 307, 323
   IdleFileCachePlugin.js97.92%100%100%97.92%71, 83, 91
   MemoryCachePlugin.js100%100%100%100%
   MemoryStore.js93.33%100%100%93.33%118, 51, 57, 60
   MemoryWithGcCachePlugin.js100%100%100%100%
   ResolverCachePlugin.js100%100%100%100%
   getLazyHashedEtag.js100%100%100%100%
   mergeEtags.js100%100%100%100%
lib/cache/format
   CacheIndex.js100%100%100%100%
   CacheSegment.js100%100%100%100%
   SegmentManager.js96.85%100%100%96.85%126, 130, 144, 236
lib/config
   browserslistTargetHandler.js100%100%100%100%
   defaults.js99.15%100%100%99.15%1342–1344, 1352, 271, 274, 279, 283, 472
   normalization.js99%100%100%99%191–192, 258, 273
   target.js100%100%100%100%
lib/container
   ContainerEntryDependency.js100%100%100%100%
   ContainerEntryModule.js100%100%100%100%
   ContainerEntryModuleFactory.js100%100%100%100%
   ContainerExposedDependency.js100%100%100%100%
   ContainerPlugin.js100%100%100%100%
   ContainerReferencePlugin.js100%100%100%100%
   FallbackDependency.js100%100%100%100%
   FallbackItemDependency.js100%100%100%100%
   FallbackModule.js100%100%100%100%
   FallbackModuleFactory.js100%100%100%100%
   HoistContainerReferencesPlugin.js100%100%100%100%
   ModuleFederationPlugin.js100%100%100%100%
   RemoteModule.js100%100%100%100%
   RemoteRuntimeModule.js100%100%100%100%
   RemoteToExternalDependency.js100%100%100%100%
   options.js100%100%100%100%
lib/css
   CssGenerator.js99.55%100%100%99.55%201, 99
   CssInjectStyleRuntimeModule.js100%100%100%100%
   CssLoadingRuntimeModule.js100%100%100%100%
   

@codecov

codecov Bot commented May 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.71222% with 162 lines in your changes missing coverage. Please review.
✅ Project coverage is 36.23%. Comparing base (46f1d0f) to head (26efab0).
⚠️ Report is 83 commits behind head on main.

Files with missing lines Patch % Lines
lib/serialization/Lazy.js 42.62% 35 Missing ⚠️
lib/serialization/Encoder.js 87.27% 28 Missing ⚠️
lib/serialization/builtins.js 32.43% 25 Missing ⚠️
lib/serialization/TypeRegistry.js 67.21% 20 Missing ⚠️
lib/serialization/FileStore.js 82.52% 18 Missing ⚠️
lib/serialization/Writer.js 78.84% 11 Missing ⚠️
lib/serialization/Decoder.js 93.52% 9 Missing ⚠️
lib/serialization/Reader.js 86.04% 6 Missing ⚠️
lib/util/serialization.js 73.33% 4 Missing ⚠️
lib/util/internalSerializables.js 0.00% 3 Missing ⚠️
... and 2 more

❌ Your patch check has failed because the patch coverage (78.71%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (46f1d0f) and HEAD (26efab0). Click for more details.

HEAD has 22 uploads less than BASE
Flag BASE (46f1d0f) HEAD (26efab0)
integration 20 0
test262 2 0
Additional details and impacted files
@@             Coverage Diff             @@
##             main   #20970       +/-   ##
===========================================
- Coverage   91.12%   36.23%   -54.90%     
===========================================
  Files         570      419      -151     
  Lines       57747    47616    -10131     
  Branches    15458    13049     -2409     
===========================================
- Hits        52622    17252    -35370     
- Misses       5125    30364    +25239     
Flag Coverage Δ
integration ?
test262 ?
unit 36.23% <78.71%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed-hq

codspeed-hq Bot commented May 16, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 50.31%

⚡ 3 improved benchmarks
❌ 9 regressed benchmarks
✅ 132 untouched benchmarks
⏩ 72 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Memory benchmark "cache-filesystem", scenario '{"name":"mode-production","mode":"production"}' 3.8 MB 9.3 MB -59.44%
Memory benchmark "context-esm", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 256.8 KB 662.6 KB -61.24%
Memory benchmark "future-defaults", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 143.7 KB 281.2 KB -48.87%
Memory benchmark "side-effects-reexport", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 859.5 KB 401.8 KB ×2.1
Memory benchmark "cache-filesystem", scenario '{"name":"mode-development","mode":"development"}' 25.9 MB 38 MB -31.75%
Memory benchmark "asset-modules-source", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 398.6 KB 3,718.4 KB -89.28%
Memory benchmark "cache-filesystem", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 193.6 KB 399 KB -51.47%
Memory benchmark "many-modules-esm", scenario '{"name":"mode-development","mode":"development"}' 1.3 MB 1.1 MB +23.24%
Memory benchmark "asset-modules-inline", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 228.6 KB 1,195 KB -80.87%
Memory benchmark "many-chunks-commonjs", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 168.7 KB 353.1 KB -52.23%
Memory benchmark "concatenate-modules", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' 137.7 KB 520.9 KB -73.56%
Memory benchmark "wasm-modules-sync", scenario '{"name":"mode-production","mode":"production"}' 7.5 MB 6 MB +24.71%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing test-different-caching-system (26efab0) with main (3032402)

Open in CodSpeed

Footnotes

  1. 72 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@avivkeller avivkeller marked this pull request as ready for review May 22, 2026 18:25
Copilot AI review requested due to automatic review settings May 22, 2026 18:25
@avivkeller avivkeller changed the title [WIP!] feat(caching): new serialization format feat(caching): new serialization format May 22, 2026
@avivkeller

Copy link
Copy Markdown
Member Author

It's hard to get a good benchmark until #20972 lands

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR replaces webpack’s persistent filesystem cache implementation and serialization stack with a new compact binary format (Encoder/Decoder + FileStore) and a segmented on-disk cache strategy (DiskCacheStrategy + cache index/segments), updating existing serializable classes and tests accordingly.

Changes:

  • Introduce a new binary serialization system (lib/serialization/*) with type registration via TypeRegistry.
  • Replace the pack-file filesystem cache with a new disk cache layout (DiskCacheStrategy, cache index + immutable segments).
  • Update core webpack classes to use the new Encoder/Decoder serializer context API and add focused unit tests for the new serializer behavior.

Reviewed changes

Copilot reviewed 137 out of 138 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
.changeset/new-cache-serialization-format.md Marks the cache format/serialization change as a major release note.
lib/AsyncDependenciesBlock.js Updates serializer context typedefs to Decoder/Encoder.
lib/ContextModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/DependenciesBlock.js Updates serializer context typedefs to Decoder/Encoder.
lib/Dependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/ExportsInfo.js Updates serializer context typedefs to Decoder/Encoder.
lib/ExternalModule.js Updates serializer context typedefs and serializer registration typing docs.
lib/FileSystemInfo.js Updates serializer context typedefs to Decoder/Encoder.
lib/InitFragment.js Updates serializer context typedefs to Decoder/Encoder.
lib/Module.js Updates serializer context typedefs to Decoder/Encoder.
lib/NormalModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/RawModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/WebpackOptionsApply.js Switches filesystem cache strategy from PackFileCacheStrategy to DiskCacheStrategy.
lib/asset/RawDataUrlModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/cache/DiskCacheStrategy.js Adds new segmented on-disk cache strategy implementation.
lib/cache/IdleFileCachePlugin.js Updates strategy type reference to DiskCacheStrategy.
lib/cache/MemoryCachePlugin.js Refactors to use shared MemoryStore helper.
lib/cache/MemoryStore.js Adds reusable in-memory cache store with optional generational GC.
lib/cache/MemoryWithGcCachePlugin.js Refactors to delegate memory GC behavior to MemoryStore.
lib/cache/ResolverCachePlugin.js Updates serializer context typedefs to Decoder/Encoder.
lib/cache/format/CacheIndex.js Adds serializable cache index + entry/segment metadata types.
lib/cache/format/CacheSegment.js Adds serializable segment container for immutable segment payloads.
lib/cache/format/SegmentManager.js Adds segment persistence/loading, GC, compaction, and orphan sweeping.
lib/container/ContainerEntryModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/container/ContainerExposedDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/container/FallbackDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/container/FallbackModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/container/RemoteModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/css/CssModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/AMDDefineDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/AMDRequireArrayDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/AMDRequireContextDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/AMDRequireDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CachedConstDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CommonJsExportRequireDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CommonJsExportsDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CommonJsFullRequireDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CommonJsRequireContextDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CommonJsRequireDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CommonJsSelfReferenceDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ConstDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ContextDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ContextElementDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CreateScriptUrlDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CssIcssExportDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CssIcssImportDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CssIcssSymbolDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CssImportDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/CssUrlDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/DllEntryDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ExportsInfoDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ExternalModuleDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ExternalModuleInitFragment.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ExternalModuleInitFragmentDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HarmonyAcceptDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HarmonyEvaluatedImportSpecifierDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HarmonyExportExpressionDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HarmonyExportHeaderDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HarmonyExportImportedSpecifierDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HarmonyExportSpecifierDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HarmonyImportDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HarmonyImportSpecifierDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HtmlScriptSrcDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/HtmlSourceDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ImportContextDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ImportDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/JsonExportsDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/LocalModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/LocalModuleDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ModuleDecoratorDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ModuleDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ModuleInitFragmentDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/ProvidedDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/PureExpressionDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/RequireEnsureDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/RequireHeaderDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/RequireResolveContextDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/RequireResolveHeaderDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/RuntimeRequirementsDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/StaticExportsDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/URLContextDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/URLDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/UnsupportedDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/WebAssemblyExportImportedDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/WebAssemblyImportDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dependencies/WorkerDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/dll/DelegatedModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/dll/DllModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/errors/HookWebpackError.js Updates serializer context typedefs to Decoder/Encoder.
lib/errors/JSONParseError.js Updates serializer context typedefs to Decoder/Encoder.
lib/errors/ModuleBuildError.js Updates serializer context typedefs to Decoder/Encoder.
lib/errors/ModuleError.js Updates serializer context typedefs to Decoder/Encoder.
lib/errors/ModuleParseError.js Updates serializer context typedefs to Decoder/Encoder.
lib/errors/ModuleWarning.js Updates serializer context typedefs to Decoder/Encoder.
lib/errors/WebpackError.js Updates serializer context typedefs to Decoder/Encoder.
lib/index.js Updates serializer context typedefs to Decoder/Encoder.
lib/json/JsonData.js Keeps external serializer registration, now backed by new util/serialization API.
lib/optimize/ConcatenatedModule.js Updates serializer context typedef to Decoder.
lib/serialization/AggregateErrorSerializer.js Removes legacy AggregateError serializer (replaced by builtins codec).
lib/serialization/ArraySerializer.js Removes legacy array serializer middleware.
lib/serialization/DateObjectSerializer.js Removes legacy Date serializer (now native tag in binary format).
lib/serialization/Decoder.js Adds binary decoder implementation.
lib/serialization/Encoder.js Adds binary encoder implementation.
lib/serialization/ErrorObjectSerializer.js Removes legacy Error serializer (replaced by builtins codec).
lib/serialization/FileMiddleware.js Removes legacy pack-file middleware implementation.
lib/serialization/FileStore.js Adds file-based store supporting compression and separate values.
lib/serialization/Lazy.js Adds shared lazy value helpers (create/is/serialize/deserialize).
lib/serialization/MapObjectSerializer.js Removes legacy Map serializer (now native tag in binary format).
lib/serialization/NullPrototypeObjectSerializer.js Removes legacy null-proto object serializer (now native tag).
lib/serialization/PlainObjectSerializer.js Removes legacy plain-object serializer (now native tag).
lib/serialization/Reader.js Adds binary reader implementation.
lib/serialization/RegExpObjectSerializer.js Removes legacy RegExp serializer (now native tag).
lib/serialization/Serializer.js Replaces middleware pipeline with Encoder/Decoder-based implementation.
lib/serialization/SerializerMiddleware.js Removes legacy middleware base (replaced by direct binary format).
lib/serialization/SetObjectSerializer.js Removes legacy Set serializer (now native tag).
lib/serialization/SingleItemMiddleware.js Removes legacy single-item middleware.
lib/serialization/TypeRegistry.js Adds codec registry + lazy loader mechanism for serializable types.
lib/serialization/Writer.js Adds binary writer implementation.
lib/serialization/builtins.js Adds built-in codecs for Error types (and AggregateError when available).
lib/serialization/format.js Defines the binary serialization tag format constants.
lib/serialization/types.js Removes legacy serialization type typedefs file.
lib/sharing/ConsumeSharedModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/sharing/ProvideSharedDependency.js Updates serializer context typedefs to Decoder/Encoder.
lib/sharing/ProvideSharedModule.js Updates serializer context typedefs to Decoder/Encoder.
lib/stats/DefaultStatsFactoryPlugin.js Replaces typedef imports with local structural error typedefs.
lib/util/LazySet.js Updates serializer context typedefs in docs to Decoder/Encoder.
lib/util/internalSerializables.js Updates internal serializables mapping for new cache/format classes.
lib/util/makeSerializable.js Reworks makeSerializable to register codecs via TypeRegistry.
lib/util/registerExternalSerializer.js Updates serializer context typedefs to Decoder/Encoder.
lib/util/serialization.js Reworks public serialization utilities to use TypeRegistry + new Serializer/FileStore.
lib/wasm-async/AsyncWebAssemblyModulesPlugin.js Updates serializer context typedefs to Decoder/Encoder.
test/BinaryMiddleware.unittest.js Removes legacy BinaryMiddleware unit tests.
test/Serializer.unittest.js Adds unit tests for new Serializer, lazy values, separate files, and references.
tooling/print-cache-file.js Updates tooling to deserialize with the new file serializer and print results.
types.d.ts Updates public type declarations for new serialization API and cache strategy.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +310 to +320
return this._getIndex()
.then(async (index) => {
const entry = index.entries.get(identifier);
if (!entry) return undefined;
if (entry.etag !== stringEtag) return null;
entry.lastAccess = Date.now();
const segment = await this.segmentManager.loadSegment(
index,
entry.segmentId
);
return segment.get(identifier);
Comment on lines +189 to +199
const oldMap = await this.loadSegment(index, id);
/** @type {Map<string, { etag: string | null, data: Data }>} */
const fresh = new Map();
for (const identifier of live) {
const entry = index.entries.get(identifier);
if (!entry) continue;
fresh.set(identifier, {
etag: entry.etag,
data: oldMap.get(identifier)
});
}
Comment thread lib/util/makeSerializable.js
@xiaoxiaojx

Copy link
Copy Markdown
Member

Ran some local benchmarks comparing the old (ObjectMiddleware + BinaryMiddleware pipeline) vs new (Encoder/Decoder) serialization on representative webpack-like data (5000 modules, 15000 dependencies, realistic path lengths and repeated type strings).

Deserialization performance concern

                 Serialize    Deserialize    Size
Old pipeline      59.0 ms       29.6 ms     4198 KB
New Encoder       37.2 ms       31.2 ms     4351 KB

Serialization is ~1.6x faster — the middleware pipeline elimination clearly pays off. However, deserialization shows a slight regression in this test shape, and in other data shapes (more nested objects, Sets) the gap widened to ~1.5x slower for the new format. The whole point of persistent cache is to speed up subsequent builds — deserialization (cache restore) runs on every cold start, while serialization only runs when content changes. A regression on the read path directly undermines the primary purpose of the cache system.

The root cause is architectural: the old BinaryMiddleware batches consecutive same-type values (N nulls → 2 bytes via RLE, N booleans → bit-packed, N integers → single header + N values), which gives the CPU predictable linear reads. The new Encoder dispatches per-value via tag bytes, which introduces more branches per value decoded.

It would be great to see deserialization benchmarks on real-world projects (e.g. the webpack codebase itself, or a large app with thousands of modules) to quantify the actual impact on cold-start cache restore time.

The architectural simplification (4-layer middleware → single-pass Encoder/Decoder) and string deduplication are solid improvements — the deserialization path is the main thing worth investigating before this lands.

@alexander-akait

Copy link
Copy Markdown
Member

@avivkeller We can't ship it in webpack@5, it is a breaking change, also a lot of changes in one PR, I can't review and validate, for such changes we need discussion firstly, I don't know architecture and to be honestly from this logic we bring nothing expect a lot of breaking changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants