Implement Vitest testing environment for Miniflare 3 - #4795

Closes all tickets in RM-17030 and RM-17070. Ref #4490.

What this PR solves / how to test:

This PR reintroduces Miniflare 2's Vitest unit testing environment to Miniflare 3! 🎉

Miniflare 3 runs your code inside workerd, rather than Node. This means we can no longer access Workers runtime APIs directly inside Node, and instead need to run tests inside a workerd process.
We do this by implementing a Vitest Custom Pool, rather than a Test Environment. Pools can completely customise how test are run, whereas environments can only customise the global scope. Many thanks to @sheremet-va for implementing custom pools, and their continued support throughout this project.
Our pool starts Miniflare instances running the regular Vitest worker script, and uses WebSockets rather than inter-thread/process communication for message passing.

This worker script expects to run inside a Node-like environment. To support this, we've implemented a "module fallback" service that provides Node-like disk-backed module resolution for require()s and import statements in workerd. Many thanks to @jasnell for implementing the fallback service API and unsafe-eval bindings this project also depends on.

Right now, the pool only supports vitest@1.1.3. I'll upgrade to the latest Vitest version soon.

Features

🟢 Implemented, 🟡 Planned, 🔴 Not Planned

• 🟢 Unit testing any function in a Worker
• 🟢 Calling fetch(), scheduled() and queue() exported handlers directly
• 🟢 Waiting until waitUntil()ed Promises have settled
• 🟢 Integration testing of Workers via service bindings
• 🟢 Multi-worker support
• 🟢 Accessing bindings, with support for seeding data
• 🟢 Isolated per-test storage
• 🟢 Directly calling Durable Object instance methods
• 🟢 Directly accessing Durable Object state
• 🟢 Immediately executing Durable Object alarms
• 🟢 Listing Durable Objects
• 🟢 Mocking outbound fetch() requests
• 🟡 Testing Pages functions
• 🟡 Easily apply D1 migrations
• Provided by Vitest:
  • 🟢 Fast re-runs with HMR
  • 🟢 Custom Reporters, Vitest UI, IDE integrations
  • 🟢 Snapshots
  • 🟢 Fake Timers
  • 🟢 Function Mocking
  • 🟡 Module Mocking
  • 🟡 Step Through Debugging of Tests
  • 🟡 Coverage
  • 🔴 Custom environments and runners inside workerd
• 🔴 Unit testing service workers
• 🔴 Jest support

Changes from vitest-environment-miniflare

• Unit testing service workers is now unsupported
• Only vitest@1.1.3 is currently supported
• The package is now @cloudflare/vitest-pool-workers
• environment: "miniflare" is now pool: "@cloudflare/vitest-pool-workers" in Vitest config
• environmentOptions: { ... } is now poolOptions: { workers: { miniflare: { ... } } } in Vitest config
• Configuration is no longer loaded from wrangler.toml by default
• getMiniflareBindings() is now import { env } from "cloudflare:test"
• getMiniflareDurableObjectStorage() is now import { runInDurableObject } from "cloudflare:test", then runInDurableObject(stub, (instance, state) => { doSomethingWith(state.storage); })
• getMiniflareDurableObjectState() is now import { runInDurableObject } from "cloudflare:test", then runInDurableObject(stub, (instance, state) => { doSomethingWith(state); })
• getMiniflareDurableObjectInstance() is now import { runInDurableObject } from "cloudflare:test", then runInDurableObject(stub, (instance) => { doSomethingWith(instance); })
• runWithMiniflareDurableObjectGates(doSomething) is now import { runInDurableObject } from "cloudflare:test", then runInDurableObject(stub, doSomething)
• getMiniflareFetchMock() is now import { fetchMock } from "cloudflare:test"
• new ExecutionContext() is now import { createExecutionContext } from "cloudflare:test"
• getMiniflareWaitUntil() is now import { getWaitUntil } from "cloudflare:test"
• flushMiniflareDurableObjectAlarms() is now import { runDurableObjectAlarm } from "cloudflare:test", then runDurableObjectAlarm(stub)
• getMiniflareDurableObjectIds() is now import { listDurableObjectIds } from "cloudflare:test"

Basic Config API Docs

// vitest.config.ts
import fs from "node:fs/promises";
import { defineWorkersPoolOptions, kCurrentWorker } from "@cloudflare/vitest-pool-workers/config";
import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    pool: "@cloudflare/vitest-pool-workers",
    poolOptions: {
      workers: defineWorkersPoolOptions({
        /**
         * Entrypoint to worker run in the same isolate/context as tests.
         * Only required if `kCurrentWorker` `serviceBindings` defined,
         * or Durable Objects without explicit `scriptName` used.
         * Note this goes through Vite transforms and can be TypeScript.
         * Note also `import module from "src/index.ts"` inside tests
         * gives exactly the same `module` instance as it used internally
         * for self-service and Durable Object bindings.
         * @optional
         */
        main: "src/index.ts",
        /**
         * Enables per-test isolated storage. If enabled, any writes to storage
         * performed in a test will be undone at the end of the test. The test
         * storage environment is copied from the containing suite, meaning
         * `beforeAll()` hooks can be used to seed data. If this is disabled,
         * all tests will share the same storage.
         * TODO: decide if we want to allow this to be disabled
         * @default false
         */
        isolatedStorage: true,
        /**
         * Runs all tests in this project serially in the same worker, using
         * the same module cache. This can significantly speed up tests if
         * you've got lots of small test files.
         * @default false
         */
        singleWorker: true,
        /**
         * Called at the start of each test run with the same `env` object as
         * `import { env } from "cloudflare:test"` inside tests. This function
         * will always be called with nothing stored in any of the bindings.
         * This function may be called multiple times per test run if
         * `singleWorker` is `false` and `isolatedStorage` is `true`.
         * @optional
         */
        async setupEnvironment(env) {
          const value = await fs.readFile("value.txt", "utf8");
          await env.TEST_NAMESPACE.put("key", value);
        },
        /**
         * Miniflare `WorkerOptions` for configuring the test environment.
         * Note configuration isn't automatically loaded from `wrangler.toml`,
         * so bindings will need to be specified here.
         * @optional
         */
        miniflare: {
          // `rootPath` is implicitly set to the dirname of the `vitest.config.ts`
          // file here. All path-valued options are resolved relative to the
          // nearest `rootPath`.
          kvNamespaces: ["TEST_NAMESPACE"],
          serviceBindings: {
            // `kCurrentWorker` here binds to the `main` worker running in the
            // test runner's isolate.
            SELF: kCurrentWorker,
            // `other` binds to the auxiliary worker with `name: "other"`
            OTHER: "other",
          },
          // If no `scriptName` is specified in the Durable Object designator,
          // it binds to the `main` worker running in the test runner's isolate,
          // and allows Durable Object test helpers to be used with stubs from
          // the namespace.
          durableObjects: { OBJECT: "TestObject" }
          workers: [
            {
              rootPath: "other-src",
              name: "other",
              modules: true,
              // Resolves to `${dirname(".../vitest.config.ts")}/other-src/index.mjs`
              scriptPath: "index.mjs"
            }
          ]
        },
      }),
    },
  },
});

Basic cloudflare:test API Docs

Miniflare Changes

This PR includes a bunch of changes in Miniflare to support the custom Vitest pool.
These changes are contained within commits starting with [miniflare].
User facing changes to non unsafe* options/methods have changesets.

• Add support for the module fallback service
• Return non-WebSocket responses for failed WebSocket upgrading fetch()es
• Export formatZodError() method
• Add support for "sticky" blobs that aren't garbage collected
• Add support for getting persistence paths
• Add support for "colo-local" ephemeral Durable Objects
• Bind this to Miniflare instance in function-valued custom services
• Use z.input() for user facing options types
• Allow easy service bindings to current worker with kCurrentWorker symbol
• Add support for rootPaths
• Fix DurableObjectStub detection when Durable Object JS RPC is enabled

Maintainer Setup Instructions

1. Make sure you're using pnpm@8 (lower versions don't correctly handle the vitest peer-dependency requirement)
2. Run pnpm install in the monorepo root
3. Run pnpm build in the monorepo root
4. Run pnpm exec vitest --version in packages/vitest-pool-workers and verify this outputs vitest/1.1.3
5. Run pnpm test in packages/vitest-pool-workers
6. Play around in packages/vitest-pool-workers/test

Pre-Release Setup Instructions

Clone https://github.com/mrbbot/vitest-pool-workers-prerelease-getting-started and follow the instructions in the README.

Reviewing Guidance

This is a pretty chunky PR. I've done my best to keep individual features to separate commits, but some later commits are fixups to earlier ones. Most commits have descriptions explaining what they're adding/changing. Very happy to walkthrough how everything fits together if that would be helpful too, but as an outline:

• packages/vitest-pool-workers
  • src
    • config: entrypoint for the @cloudflare/vitest-pool-workers/config sub-export
    • mock-agent: entrypoint for cloudflare:mock-agent worker library, subset of undici required to support MockAgent
    • pool: code running inside Node.js
      • config.ts: schemas for poolOptions.workers
      • index.ts: entrypoint for the @cloudflare/vitest-pool-workers package
      • loopback.ts: custom service for implementing snapshot storage/stacked storage
      • module-fallback.ts: fallback service implementation providing Node-like import/require module resolution
    • worker: code running inside workerd
      • lib: polyfills for modules unsupported by workerd, files in this directory map directly to imports inside workerd (i.e. lib/cloudflare/test.ts becomes cloudflare:test)
      • durable-objects.ts: Durable Object test helpers (runInDurableObject(), runDurableObjectAlarm())
      • env.ts: environment test helpers (env)
      • events.ts: exported handler test helpers (createExecutionContext(), getWaitUntil(), createScheduledController(), getScheduledResult(), createMessageBatch(), getQueueResult())
      • fetch-mock.ts: fetch mocking test helpers (fetchMock)
      • import.ts: code for importing arbitrary modules in a Worker using Vite's pipeline, mainly used for importing the configured main module
      • index.ts: entrypoint for the Worker that gets executed inside workerd to run tests, contains the Durable Object that actually imports the Vitest runner to run tests
  • test
    • basic: tests built-in Vitest functionality
    • kv: tests basic functionality of all features
    • stacked: stress test for isolated storage
    • cloudflare-test.d.ts: user facing cloudflare:test types, should probably be generated automatically

Right now, we have a few example tests in packages/vitest-pool-workers/test. I'm planning to add a wider suite of tests here soon, in addition to some "meta-tests" that verify things like snapshots work correctly (tests fail when snapshot is incorrect, snapshots can be updated, etc.).

Changesets have only been included for the miniflare package. We're not quite ready to release @cloudflare/vitest-pool-workers yet, so it's been marked private: true, and won't be included in the regular release process.

Author has addressed the following:

• Tests
  • Included
  • Not necessary because:
• Changeset (Changeset guidelines)
  • Included (for Miniflare changes only, not quite ready to release @cloudflare/vitest-pool-workers yet)
  • Not necessary because:
• Associated docs
  • Issue(s)/PR(s):
  • Not necessary because: not quite ready to release @cloudflare/vitest-pool-workers yet, upcoming meeting with PCX to discuss

Note for PR author:

We want to celebrate and highlight awesome PR review! If you think this PR received a particularly high-caliber review, please assign it the label highlight pr review so future reviewers can take inspiration and learn from it.