Enable FUSE in workerd's local-dev Docker container client#6596
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
38469d2 to
b7e40b2
Compare
| struct DeviceMapping { | ||
| pathOnHost @0 :Text; | ||
| pathInContainer @1 :Text; | ||
| cgroupPermissions @2 :Text; | ||
| pathOnHost @0 :Text $Json.name("PathOnHost"); | ||
| pathInContainer @1 :Text $Json.name("PathInContainer"); | ||
| cgroupPermissions @2 :Text $Json.name("CgroupPermissions"); | ||
| } |
There was a problem hiding this comment.
Docker's API uses PascalCase keys for these fields (PathOnHost / PathInContainer / CgroupPermissions). Without these $Json.name annotations, capnp's JSON codec emits the source field names in camelCase, which Docker silently ignores
b7e40b2 to
5b7d5cc
Compare
|
I have read the CLA Document and I hereby sign the CLA |
|
recheck |
The reproduction's Docker proxy previously injected Privileged=true alongside CapAdd/Devices/SecurityOpt. Privileged was redundant in practice (it masked whether the other three were doing anything), and made it look like the proxy and the workerd PR were doing different things. Narrowed the proxy to only the three HostConfig fields that the upstream PR (cloudflare/workerd#6596) enables: CapAdd=[SYS_ADMIN], Devices=[/dev/fuse], SecurityOpt=[apparmor:unconfined]. All 3 vitest scenarios still pass. The proxy and the patched-workerd now both make FUSE work with the exact same minimum capability set.
|
/bonk review I think this is likely fine so long as it doesn't require running with |
There was a problem hiding this comment.
Thanks for the contribution! The docker-api.capnp fix is clearly correct and needed. The FUSE enablement logic works but I have a few concerns about the scope and unconditional nature of the changes.
Summary:
- The
$Json.nameannotation fix onDeviceMappingis a genuine bug fix — without it the fields serialize as camelCase and Docker silently ignores them. Good catch. - Unconditionally adding
SYS_ADMIN+/dev/fuse+apparmor:unconfinedto every container is a broad change. Most containers don't need FUSE, and this widens the attack surface for all of them. Even in local dev, this adds capabilities that could mask bugs or create surprising behavior for users who aren't using FUSE. - The
apparmor:unconfinedsetting is Linux-specific. On macOS (Docker Desktop with a Linux VM), AppArmor typically isn't active, so this is harmless but unnecessary. Not a blocker, but worth noting.
See inline comments for specifics.
| // Docker doesn't grant FUSE access by default — enable the minimum permissions for it. | ||
| { | ||
| auto capAdd = hostConfig.initCapAdd(1); | ||
| capAdd.set(0, "SYS_ADMIN"); | ||
| } | ||
| { | ||
| auto devices = hostConfig.initDevices(1); | ||
| auto fuseDev = devices[0]; | ||
| fuseDev.setPathOnHost("/dev/fuse"); | ||
| fuseDev.setPathInContainer("/dev/fuse"); | ||
| fuseDev.setCgroupPermissions("rwm"); | ||
| } | ||
| { | ||
| auto securityOpt = hostConfig.initSecurityOpt(1); | ||
| securityOpt.set(0, "apparmor:unconfined"); | ||
| } |
There was a problem hiding this comment.
This unconditionally grants SYS_ADMIN, maps /dev/fuse, and disables AppArmor for every container created via local dev, regardless of whether the user's workload uses FUSE. CAP_SYS_ADMIN is one of the most powerful Linux capabilities — it covers mount(), setns(), unshare(), bpf(), and more. Granting it by default to all local containers when only a small subset need FUSE seems overly broad.
A few options to consider:
- Opt-in flag: Gate this behind a configuration option (the approach mentioned in 🐛 Bug Report — Runtime APIs: No privileged mode option for Containers local development #5609). Users who need FUSE explicitly enable it.
- Check for
/dev/fuseon the host: Only add the device mapping and capabilities if/dev/fuseactually exists on the host. This is a lighter heuristic. - At minimum, add a comment explaining the security trade-off so future readers understand why every local container gets
SYS_ADMIN.
I'd defer to the maintainers on which approach they prefer, but option 1 seems safest for a first pass — it avoids surprising users who don't need FUSE with elevated container privileges.
| } | ||
| { | ||
| auto securityOpt = hostConfig.initSecurityOpt(1); | ||
| securityOpt.set(0, "apparmor:unconfined"); |
There was a problem hiding this comment.
Nit: apparmor:unconfined is a no-op on macOS (Docker Desktop runs a Linux VM that may or may not have AppArmor). Not a problem — Docker just ignores unknown security options — but worth a brief comment noting it's Linux-specific, similar to how gatewayForPlatform (line 873) documents its macOS behavior.
| // Docker doesn't grant FUSE access by default — enable the minimum permissions for it. | ||
| { | ||
| auto capAdd = hostConfig.initCapAdd(1); | ||
| capAdd.set(0, "SYS_ADMIN"); |
There was a problem hiding this comment.
Since the sidecar container (line 1976) already uses initCapAdd(1) for NET_ADMIN, this follows the existing pattern well. However, if FUSE support is eventually gated behind an opt-in, this block should be conditional. Even if it stays unconditional, consider adding a brief rationale comment explaining why SYS_ADMIN specifically (i.e., FUSE needs the mount() syscall) so future readers don't mistake it for an oversight.
|
Review posted successfully. Here's a summary of the feedback I left: docker-api.capnp fix (lines 35-37): The FUSE enablement in container-client.c++ (lines 1701-1716): The main concern is that
I also noted that |
67aeb3b to
c12a5d6
Compare
|
@gpanders Is there anything you need me to do here? Thanks! |
|
I think this needs to be opt-in, at least for now, but I'm unsure what the best method is to do that. I've asked the @cloudflare/wrangler team for their input. |
emily-shen
left a comment
There was a problem hiding this comment.
so to make this opt-in during dev, you'll need to add an option (e.g. allowPrivileged, but i have no real preference on naming) to ContainerOptions in workerd.capnp.
then you'll have to make another PR to https://github.com/cloudflare/workers-sdk that sets that adds a cli flag and a top-level config option (we also need that so vite dev users can pass it in), thread that through miniflare and into where we set the workerd options.
Happy to put up a PR for the workers-sdk side if you want
c12a5d6 to
4f98512
Compare
…ers for FUSE in local dev Containers using FUSE work in production but break in `wrangler dev` because workerd doesn't grant CAP_SYS_ADMIN, mount /dev/fuse, or relax AppArmor on the container it creates. Adds an opt-in dev config flag — `dev.privileged_containers` in wrangler.json, `--privileged-containers` on the CLI — that, when set, has miniflare ask workerd to launch local containers with those three permissions. Off by default; only takes effect when the user explicitly opts in. Pairs with the workerd change in cloudflare/workerd#6596, which adds the matching `allowPrivileged` field to ContainerOptions and gates the FUSE injection on it.
|
@emily-shen I am working on a draft PR for Quick note @gpanders I had since we now are passing something like This PR passes Some light upsides if we do flip it: simpler API (one boolean instead of three coupled fields), and more future-proof if anything needs more kernel access later. Trade-off is broader privilege escalation, but only in local dev. I don't have strong opinions here, just wanted to flag it. |
|
The "privileged" flag is meant to signal "the container will run with the ability to do things it normally couldn't do" which is indeed the case with the SYS_ADMIN capability. It does not (need to) correspond to the literal We can always make the "privileged" flag/option do more things in the future (including setting |
…ers for FUSE in local dev Containers using FUSE work in production but break in `wrangler dev` because workerd doesn't grant CAP_SYS_ADMIN, mount /dev/fuse, or relax AppArmor on the container it creates. Adds an opt-in dev config flag — `dev.privileged_containers` in wrangler.json, `--privileged-containers` on the CLI — that, when set, has miniflare ask workerd to launch local containers with those three permissions. Off by default; only takes effect when the user explicitly opts in. Pairs with the workerd change in cloudflare/workerd#6596, which adds the matching `allowPrivileged` field to ContainerOptions and gates the FUSE injection on it.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6596 +/- ##
==========================================
- Coverage 70.85% 66.57% -4.28%
==========================================
Files 438 402 -36
Lines 123642 115879 -7763
Branches 19455 19397 -58
==========================================
- Hits 87604 77145 -10459
- Misses 24514 27163 +2649
- Partials 11524 11571 +47 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
4f98512 to
b67c735
Compare
…tainers
Containers using FUSE work in production but break in `wrangler dev`
because workerd doesn't grant CAP_SYS_ADMIN, mount /dev/fuse, or relax
AppArmor on the container it creates. Without those three knobs the
mount("fuse", ...) syscall inside the container fails.
Add a new boolean to ContainerOptions in workerd.capnp, off by default:
struct ContainerOptions {
imageName @0 :Text;
allowPrivileged @1 :Bool = false; # NEW
}
When set on a DurableObjectNamespace's container config, ContainerClient
adds the minimum HostConfig fields needed for FUSE to the
POST /containers/create payload:
- CapAdd: ["SYS_ADMIN"] (for the mount() syscall)
- Devices: [/dev/fuse] (so the kernel has something to open)
- SecurityOpt: ["apparmor:unconfined"] (bypasses the default docker
apparmor profile's mount block)
No Privileged=true. Mirrors what Cloudflare's production container
runtime effectively provides for FUSE workers, without granting all
Linux capabilities or full /dev passthrough.
server.c++ reads the flag from the per-DO ContainerOptions config and
threads it through to the ContainerClient constructor. Off by default
means existing configs are unaffected. Only the local Docker code path
is touched; production is unaffected.
Also adds missing $Json.name annotations to docker_api::DeviceMapping
fields in docker-api.capnp so they serialize as PathOnHost /
PathInContainer / CgroupPermissions to match the Docker API. This is
load-bearing for the Devices bind above — without it, the entry
serializes as camelCase and Docker silently ignores it. Before this PR
no call site populated Devices, so the missing annotations were latent.
Follows the existing pattern of createSidecarContainer() which already
sets CapAdd=[NET_ADMIN] for its own need at container-client.c++:1958.
The matching workers-sdk PR
(cloudflare/workers-sdk#13748) adds
`dev.enable_containers_privileged_mode` and
`--enable-containers-privileged-mode` so wrangler / vite-plugin users
can opt in.
End-to-end reproduction (libc mount("fuse", ...) syscall succeeding
inside the container with the flag set, and failing without it) at
https://github.com/Ben2W/workerd-fuse-local-repro
Signed-off-by: Ben Werner <bewerner23@gmail.com>
b67c735 to
ce00b5f
Compare
…ers for FUSE in local dev Containers using FUSE work in production but break in `wrangler dev` because workerd doesn't grant CAP_SYS_ADMIN, mount /dev/fuse, or relax AppArmor on the container it creates. Adds an opt-in dev config flag — `dev.privileged_containers` in wrangler.json, `--privileged-containers` on the CLI — that, when set, has miniflare ask workerd to launch local containers with those three permissions. Off by default; only takes effect when the user explicitly opts in. Pairs with the workerd change in cloudflare/workerd#6596, which adds the matching `allowPrivileged` field to ContainerOptions and gates the FUSE injection on it.
Adds a callout to the R2 FUSE example, a new troubleshooting section in the Containers local-dev page, and a changelog entry, all explaining that FUSE in `wrangler dev` requires the new opt-in flag (`dev.privileged_containers` / `--privileged-containers`). Pairs with cloudflare/workerd#6596 and cloudflare/workers-sdk#13748. Resolves cloudflare/workerd#5609.
|
@gpanders This has been opened for a while. I added test coverage as per the Codedev report. Is there any way we can push this through? I also rebased this PR, and the related cloudflare-docs PR, and wrangler-sdk PR |
Closes #5609.
Problem
wrangler dev(via miniflare) doesn't support FUSE in locally-spawned Cloudflare Containers, even though Workers using FUSE run fine in production. This is becauseContainerClient::createContainer()POSTs to the user's local Docker daemon's/containers/createendpoint without the HostConfig fields that make/dev/fuseusable inside the container.This PR sets three HostConfig fields on the user's app container:
CapAdd=[SYS_ADMIN],Devices=[/dev/fuse],SecurityOpt=[apparmor:unconfined]. Also adds missing$Json.nameannotations toDeviceMappingindocker-api.capnpI created a reproduction repo that demonstrates using this elevated HostConfig fixes the FUSE mount when running
wrangler dev: https://github.com/Ben2W/workerd-fuse-local-reproNotes
CAP_SYS_ADMINelevates the container's permissions — it covers themount()syscall FUSE needs, alongside a range of other privileged kernel operations, and has historically been the pre-condition for container-escape CVEs (CVE-2022-0492, CVE-2022-0185). The blast radius here is narrow: this code path only runs duringwrangler devon a developer's own machine, against their own Docker daemon, with an image they've chosen. If this is still too coarse for an unconditional default, I'm happy to pivot to an opt-in design similar to what was pitched in #5609.