[wrangler/miniflare] Add privileged_containers / --privileged-containers for FUSE in local dev#13748
[wrangler/miniflare] Add privileged_containers / --privileged-containers for FUSE in local dev#13748Ben2W wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 37d5ae5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 6 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…tainers
Containers using FUSE work in production but break in `wrangler dev`
because workerd doesn't grant CAP_SYS_ADMIN, mount /dev/fuse, or relax
AppArmor on the container it creates. Without those three knobs the
mount("fuse", ...) syscall inside the container fails.
Add a new boolean to ContainerOptions in workerd.capnp, off by default:
struct ContainerOptions {
imageName @0 :Text;
allowPrivileged @1 :Bool = false; # NEW
}
When set on a DurableObjectNamespace's container config, ContainerClient
adds the minimum HostConfig fields needed for FUSE to the
POST /containers/create payload:
- CapAdd: ["SYS_ADMIN"] (for the mount() syscall)
- Devices: [/dev/fuse] (so the kernel has something to open)
- SecurityOpt: ["apparmor:unconfined"] (bypasses the default docker
apparmor profile's mount block)
No Privileged=true. Mirrors what Cloudflare's production container
runtime effectively provides for FUSE workers, without granting all
Linux capabilities or full /dev passthrough.
server.c++ reads the flag from the per-DO ContainerOptions config and
threads it through to the ContainerClient constructor. Off by default
means existing configs are unaffected. Only the local Docker code path
is touched; production is unaffected.
Also adds missing $Json.name annotations to docker_api::DeviceMapping
fields in docker-api.capnp so they serialize as PathOnHost /
PathInContainer / CgroupPermissions to match the Docker API. This is
load-bearing for the Devices bind above — without it, the entry
serializes as camelCase and Docker silently ignores it. Before this PR
no call site populated Devices, so the missing annotations were latent.
Follows the existing pattern of createSidecarContainer() which already
sets CapAdd=[NET_ADMIN] for its own need at container-client.c++:1958.
The matching workers-sdk PR
(cloudflare/workers-sdk#13748) adds
`dev.enable_containers_privileged_mode` and
`--enable-containers-privileged-mode` so wrangler / vite-plugin users
can opt in.
End-to-end reproduction (libc mount("fuse", ...) syscall succeeding
inside the container with the flag set, and failing without it) at
https://github.com/Ben2W/workerd-fuse-local-repro
Signed-off-by: Ben Werner <bewerner23@gmail.com>
| * | ||
| * @default false | ||
| */ | ||
| enable_containers_privileged_mode: boolean; |
There was a problem hiding this comment.
This name is pretty verbose. What about just privileged, enabled with wrangler dev --privileged?
Docs (and help text) will indicate what "privileged" means.
I defer to the @cloudflare/wrangler team on this, but that's my 2c.
If just privileged is too terse, then privileged_containers maybe (wrangler dev --privileged-containers).
There was a problem hiding this comment.
Good callout, --privileged-containers is 100% better than --enable-containers-privileged-mode. I don't have a strong opinion between --privileged and --privileged-containers, but --privileged-containers seems safer so I pushed that.
84f41da to
262df01
Compare
262df01 to
0bfc42c
Compare
| imageName?: string; | ||
| allowPrivileged?: boolean; | ||
| }; | ||
|
|
There was a problem hiding this comment.
Heads up: Worker_DurableObjectNamespace wasn't declaring a container field on main at all, even though miniflare was already passing container: { imageName } through to workerd at runtime. Happy to pull this out of the PR if you'd rather keep it scoped, let me know.
|
Codeowners approval required for this PR:
Show detailed file reviewers |
petebacondarwin
left a comment
There was a problem hiding this comment.
LGTM but marking as draft and blocked until cloudflare/workerd#6596 has landed and we have bumped the workerd version in workers-sdk.
| @@ -771,6 +774,14 @@ function normalizeAndValidateDev( | |||
| "boolean" | |||
| ); | |||
|
|
|||
| validateOptionalProperty( | |||
There was a problem hiding this comment.
I've just realised that the way that normalizeAndValidateDev() is structured, all these calls to validateOptionaProperty() are pointless, since we have already forced the values not to be undefined in the assignment from rawDev above. So these could just have been validateRequiredProperty().
Anyway, nothing to do with this PR.
| @@ -724,6 +726,7 @@ function normalizeAndValidateDev( | |||
| : local_protocol, | |||
| host, | |||
| enable_containers = enableContainersArg ?? true, | |||
| privileged_containers = privilegedContainersArg ?? false, | |||
There was a problem hiding this comment.
Hmm, I just realised that this means that the Wrangler config value will override any command line arg, right?
Is that what you want? Classically the ordering tends to be command line arg > env var > config value > default.
I appreciate that this is how many of the other properties are being computed too. Although notably port is validated "correctly". (It's default is set later
workers-sdk/packages/wrangler/src/api/startDevWorker/ConfigController.ts
Lines 139 to 142 in 47cf644
workers-devprod
left a comment
There was a problem hiding this comment.
Codeowners reviews satisfied
…ers for FUSE in local dev Containers using FUSE work in production but break in `wrangler dev` because workerd doesn't grant CAP_SYS_ADMIN, mount /dev/fuse, or relax AppArmor on the container it creates. Adds an opt-in dev config flag — `dev.privileged_containers` in wrangler.json, `--privileged-containers` on the CLI — that, when set, has miniflare ask workerd to launch local containers with those three permissions. Off by default; only takes effect when the user explicitly opts in. Pairs with the workerd change in cloudflare/workerd#6596, which adds the matching `allowPrivileged` field to ContainerOptions and gates the FUSE injection on it.
…tainers
Containers using FUSE work in production but break in `wrangler dev`
because workerd doesn't grant CAP_SYS_ADMIN, mount /dev/fuse, or relax
AppArmor on the container it creates. Without those three knobs the
mount("fuse", ...) syscall inside the container fails.
Add a new boolean to ContainerOptions in workerd.capnp, off by default:
struct ContainerOptions {
imageName @0 :Text;
allowPrivileged @1 :Bool = false; # NEW
}
When set on a DurableObjectNamespace's container config, ContainerClient
adds the minimum HostConfig fields needed for FUSE to the
POST /containers/create payload:
- CapAdd: ["SYS_ADMIN"] (for the mount() syscall)
- Devices: [/dev/fuse] (so the kernel has something to open)
- SecurityOpt: ["apparmor:unconfined"] (bypasses the default docker
apparmor profile's mount block)
No Privileged=true. Mirrors what Cloudflare's production container
runtime effectively provides for FUSE workers, without granting all
Linux capabilities or full /dev passthrough.
server.c++ reads the flag from the per-DO ContainerOptions config and
threads it through to the ContainerClient constructor. Off by default
means existing configs are unaffected. Only the local Docker code path
is touched; production is unaffected.
Also adds missing $Json.name annotations to docker_api::DeviceMapping
fields in docker-api.capnp so they serialize as PathOnHost /
PathInContainer / CgroupPermissions to match the Docker API. This is
load-bearing for the Devices bind above — without it, the entry
serializes as camelCase and Docker silently ignores it. Before this PR
no call site populated Devices, so the missing annotations were latent.
Follows the existing pattern of createSidecarContainer() which already
sets CapAdd=[NET_ADMIN] for its own need at container-client.c++:1958.
The matching workers-sdk PR
(cloudflare/workers-sdk#13748) adds
`dev.enable_containers_privileged_mode` and
`--enable-containers-privileged-mode` so wrangler / vite-plugin users
can opt in.
End-to-end reproduction (libc mount("fuse", ...) syscall succeeding
inside the container with the flag set, and failing without it) at
https://github.com/Ben2W/workerd-fuse-local-repro
Signed-off-by: Ben Werner <bewerner23@gmail.com>
…tainers
Containers using FUSE work in production but break in `wrangler dev`
because workerd doesn't grant CAP_SYS_ADMIN, mount /dev/fuse, or relax
AppArmor on the container it creates. Without those three knobs the
mount("fuse", ...) syscall inside the container fails.
Add a new boolean to ContainerOptions in workerd.capnp, off by default:
struct ContainerOptions {
imageName @0 :Text;
allowPrivileged @1 :Bool = false; # NEW
}
When set on a DurableObjectNamespace's container config, ContainerClient
adds the minimum HostConfig fields needed for FUSE to the
POST /containers/create payload:
- CapAdd: ["SYS_ADMIN"] (for the mount() syscall)
- Devices: [/dev/fuse] (so the kernel has something to open)
- SecurityOpt: ["apparmor:unconfined"] (bypasses the default docker
apparmor profile's mount block)
No Privileged=true. Mirrors what Cloudflare's production container
runtime effectively provides for FUSE workers, without granting all
Linux capabilities or full /dev passthrough.
server.c++ reads the flag from the per-DO ContainerOptions config and
threads it through to the ContainerClient constructor. Off by default
means existing configs are unaffected. Only the local Docker code path
is touched; production is unaffected.
Also adds missing $Json.name annotations to docker_api::DeviceMapping
fields in docker-api.capnp so they serialize as PathOnHost /
PathInContainer / CgroupPermissions to match the Docker API. This is
load-bearing for the Devices bind above — without it, the entry
serializes as camelCase and Docker silently ignores it. Before this PR
no call site populated Devices, so the missing annotations were latent.
Follows the existing pattern of createSidecarContainer() which already
sets CapAdd=[NET_ADMIN] for its own need at container-client.c++:1958.
The matching workers-sdk PR
(cloudflare/workers-sdk#13748) adds
`dev.enable_containers_privileged_mode` and
`--enable-containers-privileged-mode` so wrangler / vite-plugin users
can opt in.
End-to-end reproduction (libc mount("fuse", ...) syscall succeeding
inside the container with the flag set, and failing without it) at
https://github.com/Ben2W/workerd-fuse-local-repro
Signed-off-by: Ben Werner <bewerner23@gmail.com>
0bfc42c to
37d5ae5
Compare
Adds a callout to the R2 FUSE example, a new troubleshooting section in the Containers local-dev page, and a changelog entry, all explaining that FUSE in `wrangler dev` requires the new opt-in flag (`dev.privileged_containers` / `--privileged-containers`). Pairs with cloudflare/workerd#6596 and cloudflare/workers-sdk#13748. Resolves cloudflare/workerd#5609.
Pairs with cloudflare/workerd#6596. Resolves cloudflare/workerd#5609.
Adds an opt-in dev config flag for FUSE in
wrangler dev:dev.privileged_containersinwrangler.json--privileged-containerson the CLIWhen set, miniflare asks workerd (via the per-DO
container.allowPrivilegedfield added in the paired PR) to launch local containers withCAP_SYS_ADMIN,/dev/fuse, and AppArmor unconfined. This gives local containers permission to mount FUSE volumes.Reproduction harness: https://github.com/Ben2W/workerd-fuse-local-repro
Docs PR: cloudflare/cloudflare-docs#30462
CC @emily-shen