I went down a rabbit hole trying to get CUDA Inter-Process Communication (IPC) working reliably in Kubernetes, without throwing security out the window. I tested four real setups, flipped every permission I could, broke things on purpose, and kept notes on what actually mattered.

This post is the write-up I wish I had at the start. It keeps all the gritty details, but tells the story like a human who has had cudaIpcOpenMemHandle() stare back with “invalid argument” at 2 a.m.

TL;DR

• hostIPC is not required if you pass IPC handles via a shared file or volume.
• DRA (Dynamic Resource Allocation, Kubernetes 1.31+) removes the need for privileged containers.
• For single-pod setups, shareProcessNamespace is a safer substitute for hostPID.
• The most secure working setup: single pod + DRA + shareProcessNamespace, with both hostPID and privileged turned off.

If you just want working YAML, jump to Configuration #4.

How the repo is laid out

This work lives in the repo: cuda-ipc-debugging. There are four self-contained examples with code and manifests:

cuda-ipc-debugging/
|-- shared-volume-example/          # #1 Multi-Pod, traditional device plugin
|-- dra-shared-gpu-example/         # #2 Multi-Pod, DRA
|-- single-pod-example/             # #3 Single Pod, traditional device plugin
`-- single-pod-dra-example/         # #4 Single Pod, DRA (best security)

Each directory has a README and runnable YAMLs. All findings below come from actually running these on Kubernetes and toggling permissions until things either worked or failed with specific errors.

A quick mental model of the knobs

• hostIPC: Lets you join the host IPC namespace. For CUDA IPC we do not need this if handles are exchanged via files.
• hostPID: Lets a container see processes in the host PID namespace. CUDA IPC cares about process visibility; without it, the consumer cannot open the producer’s handle in multi-pod setups.
• privileged: Drops most isolation. With the traditional NVIDIA device plugin, it is often used to let multiple containers touch the same physical GPU devices. It is a big security hammer.
• shareProcessNamespace: Shares a PID namespace across containers in the same pod. Crucially, this gives CUDA IPC the process visibility it needs without exposing the whole host.
• DRA: The newer Kubernetes (1.31+) way to allocate GPUs via ResourceClaims. It solves many of the device-sharing problems that used to force us into privileged mode.

With that out of the way, here is what I actually saw, configuration by configuration.

Configuration #1: Multi-Pod, traditional device plugin

Two pods (producer and consumer), a hostPath volume to drop the handle file, and the classic NVIDIA device plugin managing GPU allocation.

What worked in practice

• hostPID must be true; without it, the consumer cannot see the producer’s process context and you will hit “invalid device context”.
• hostIPC is not needed.
• privileged unfortunately is required. Without it, the two pods end up with access to different physical GPUs, and cudaIpcOpenMemHandle() returns “invalid argument”. Privileged lets both pods see the device nodes they need to coordinate on the same physical GPU.

Why privileged really matters here

The device plugin isolates GPUs per pod. I verified with strace, Bus-Id checks, and by toggling capabilities (SYS_ADMIN, MKNOD, DAC_OVERRIDE, SYS_RAWIO); none of those capabilities alone fixed it. Privileged did, because it bypasses the device isolation enough for both pods to land on the same card.

Producer pod (example)

The full manifests live in the repo: shared-volume-example/producer-pod.yaml and the matching consumer-pod.yaml.

When I would reach for this

• Older clusters without DRA.
• Quick experiments or local testing.

Security notes

This is the least secure option (privileged plus hostPID). Use it only when you must, and keep it isolated.

Configuration #2: Multi-Pod with DRA

Still two pods and a shared volume for the handle file, but GPUs are allocated via a shared ResourceClaim using Kubernetes DRA.

What changed with DRA

• hostPID still needs to be true for the producer and consumer to see each other’s process context across pods. Disabling it produced “invalid device context”.
• hostIPC continues to be unnecessary.
• privileged can be false. This is the big win. DRA gets both pods coordinated access to the same underlying GPU resources without smashing the isolation wall.

GPU selection that stays sane

I used CUDA_VISIBLE_DEVICES so each pod sees only the GPU it is meant to touch:

env:
- name: CUDA_VISIBLE_DEVICES
  value: "0"

env:
- name: CUDA_VISIBLE_DEVICES
  value: "1"

Both containers can still use straightforward code like cudaSetDevice(0) internally.

ResourceClaim (example)

Full YAML: dra-shared-gpu-example/resource-claim.yaml.

Producer pod (sketch)

Full manifest: dra-shared-gpu-example/producer-pod.yaml. The consumer pod for this scenario is adjacent in the same directory.

When I would use this

• Production pipelines that need separate pods.
• Places where privileged is a non-starter but cross-pod GPU IPC is required.

Security notes

Much better than Configuration #1. You still need hostPID, but you can keep privileged off.

Configuration #3: Single pod, traditional device plugin

One pod with two containers. The handle file lives in an emptyDir. This is where shareProcessNamespace shines.

What mattered

• You need process visibility between containers. That can be hostPID: true, or (better) shareProcessNamespace: true.
• You still need privileged: true with the traditional device plugin; otherwise you will hit “invalid argument” opening the IPC handle.
• hostIPC is not required.

Two minimal shapes that worked

Option A: hostPID

spec:
  hostIPC: false
  hostPID: true
  containers:
  - name: producer
    securityContext:
      privileged: true

Option B: shareProcessNamespace (preferred)

spec:
  hostIPC: false
  shareProcessNamespace: true
  containers:
  - name: producer
    securityContext:
      privileged: true

Why shareProcessNamespace is a big deal

It gives each container in the pod the process visibility CUDA IPC needs, without exposing host processes. In single-pod scenarios it is strictly safer than hostPID and worked reliably in my tests.

Sketch YAML

For the full pod spec, see single-pod-example/cuda-ipc-pod-concurrent.yaml.

When I would use this

• Tightly coupled stages that should colocate.
• Simpler scheduling and resource guarantees in one pod.

Security notes

Better with shareProcessNamespace than with hostPID, but still uses privileged due to the device plugin. If you can, jump to DRA.

Configuration #4: Single pod with DRA (best security)

This is the setup I recommend if you can use Kubernetes 1.31+ and the NVIDIA GPU DRA driver. One pod, two containers, emptyDir for the handle file, and DRA to coordinate GPU access.

What makes this great

• hostPID: false (default) - not needed.
• hostIPC: false (default) - not needed.
• shareProcessNamespace: true - gives the CUDA IPC process visibility, pod-scoped.
• privileged: false (default) - DRA means you do not need it.

Minimal shape

spec:
  hostIPC: false
  hostPID: false
  shareProcessNamespace: true
  containers:
  - name: producer
    securityContext:
      privileged: false
      capabilities:
        add: ["IPC_LOCK"]

GPU selection without surprises

Both containers can see two devices but pick different ones. I used environment variables for clarity:

env:
- name: CUDA_VISIBLE_DEVICES
  value: "0,1"

Then in code:

• Producer: cudaSetDevice(0) allocates and exports.
• Consumer: cudaSetDevice(1) imports and operates.

ResourceClaim and pod (example)

The full manifests are here:

• single-pod-dra-example/resource-claim.yaml
• single-pod-dra-example/single-pod-dra.yaml

When I would use this

• Production workloads in multi-tenant or regulated environments.
• Anywhere privileged is off-limits (which should be most places).

Security notes

This hits the sweet spot: no host namespaces, no privileged containers, only pod-scoped process sharing.

Cheat sheet

• Multi-Pod, traditional: hostPID=true, privileged=true, hostIPC=false.
• Multi-Pod, DRA: hostPID=true, privileged=false, hostIPC=false.
• Single pod, traditional: shareProcessNamespace=true (or hostPID=true), privileged=true, hostIPC=false.
• Single pod, DRA: shareProcessNamespace=true, privileged=false, hostPID=false, hostIPC=false.

Or, in table form:

Notes:

• When I disabled hostPID in the multi-pod cases, I consistently saw “invalid device context”.
• When I disabled privileged under the traditional device plugin, I saw “invalid argument” from cudaIpcOpenMemHandle().
• hostIPC never mattered when the handle moved through a file or volume.

Recommendations

• If you can, adopt DRA (Kubernetes 1.31+) and aim for Configuration #4. It is the cleanest and most secure.
• In single-pod scenarios, prefer shareProcessNamespace over hostPID.
• Avoid privileged unless you are stuck with the traditional device plugin and truly have no alternative.
• Treat hostPID as a last resort, and only where cross-pod visibility is unavoidable.

Resources

• NVIDIA GPU DRA Driver: https://github.com/NVIDIA/k8s-dra-driver
• Kubernetes DRA docs: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/
• Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/

This post is based on hands-on testing of all four setups. The successes and failures (and their exact errors) are from real runs, not theory. If you spot something different in your cluster, I would love to hear about it.

This article was originally published on Red Hat Developer.

NVIDIA Multi-Instance GPU (MIG) allows dynamic GPU resource allocation in Kubernetes, enabling more efficient and flexible GPU usage across different AI/ML workloads. The dynamic accelerator slicer operator in Red Hat OpenShift brings this power to developers in an on-demand, developer-friendly way.

The Challenge

Running multiple AI models with vastly different resource needs on the same GPU infrastructure creates a dilemma: do you provision for the largest model and waste resources when running smaller ones, or do you manage separate GPU pools for different model sizes? Neither approach is ideal.

The Solution

MIG turns one GPU into many independent slices, each with isolated memory and compute resources. Combined with the dynamic accelerator slicer operator in OpenShift, you get:

• Flexible allocation: Create GPU slices on-demand based on workload requirements
• Better utilization: Run seven tiny models, two medium models, or one large model on the same physical GPU
• Kubernetes-native: Managed through standard Kubernetes resources and workflows

Real-World Examples

The article demonstrates three GPU slicing scenarios using vLLM:

1. Seven tiny models (Gemma 3 270M) on 1g.5gb slices
2. Two medium models (Qwen2-7B-Instruct) on 3g.20gb slices
3. One large model (GPT-OSS 20B) on a 7g.40gb slice

Each scenario shows how the same physical GPU can be dynamically reconfigured to match the exact needs of different AI workloads, eliminating resource waste and improving overall cluster utilization.

Read the full article on Red Hat Developer: Dynamic GPU Slicing with Red Hat OpenShift and NVIDIA MIG

Distributed workloads love to act like a synchronized swim team: either everyone dives in together or someone belly flops alone. Gang scheduling is how you keep the choreography perfect. Using LeaderWorkerSet (LWS) together with the Volcano scheduler, we can see exactly how that works across a few real scenarios. The manifests and scripts in this companion sandbox — lws-gang-demo — simply give us a convenient stage.

Why Gang Scheduling Exists

Multi-pod jobs fall apart when only part of the team runs. Without coordination, Kubernetes may start one pod, leave three Pending, and your training run or inference pipeline stalls forever. Gang scheduling fixes that by treating the pods as one atomic unit. If every pod in the gang can be scheduled, they all launch together; if not, they all wait.

With LWS, we get leader/worker orchestration for stateful or distributed jobs. Pairing it with Volcano brings the “all-or-nothing” scheduling behaviour needed for reliability.

Setting the Stage

The sandbox uses a small kind cluster and a few setup steps:

1. Create the environment – scripts/setup-cluster.sh builds a four-node kind cluster, installs Volcano, installs LWS v0.7.0, and patches the LWS config so gangSchedulingManagement.schedulerProvider is set to volcano.
2. Grant permissions – manifests/setup/volcano-rbac.yaml gives the LWS controller the right to create and manage Volcano PodGroups.
3. Apply workloads – each scenario is defined in manifests/examples/*.yaml, and scripts/run-demo.sh or scripts/verify-gang-scheduling.sh guide you through them step by step.

That’s all scaffolding; the interesting part is watching what happens when pods try to schedule.

Scenario 1 – All Pods Ready? Launch Together

Manifest: manifests/examples/gang-test.yaml

• 1 leader + 3 workers, each tagged with schedulerName: volcano.
• LWS automatically creates a PodGroup with minMember = 4.
• Volcano keeps the entire set “Inqueue” until four slots open, then flips every pod to Running in one shot.

$ kubectl get podgroups -n gang-demo
NAME                      STATUS    MINMEMBER   RUNNINGS
gang-test-0-6c67cc9968    Running   4           4

Takeaway: the gang scheduler is invisible when capacity exists — everything still starts instantly, just with guardrails in place.

Scenario 2 – Resource Pinch, Zero Partial Launches

Manifest: manifests/examples/gang-constrained.yaml

• Pods request 4 CPUs each; the script taints two worker nodes so only one remains usable.
• LWS still creates the PodGroup (minMember = 4, minResources equal to the summed requests).
• Volcano sees the gang cannot fit, so every pod stays Pending. Events show the PodGroup stuck with Unschedulable until more nodes free up.
• The moment a taint is removed, Volcano schedules all four pods atomically.

$ kubectl get pods -n gang-demo -l leaderworkerset.sigs.k8s.io/name=gang-constrained
NAME                   READY   STATUS    RESTARTS   AGE
gang-constrained-0     0/1     Pending   0          24s
gang-constrained-0-1   0/1     Pending   0          24s
gang-constrained-0-2   0/1     Pending   0          24s
gang-constrained-0-3   0/1     Pending   0          24s

$ kubectl get podgroups -n gang-demo
NAME                           STATUS    MINMEMBER   RUNNINGS
<pod-group-name>              Inqueue   4           <none>

$ kubectl get events --field-selector involvedObject.kind=PodGroup -n gang-demo
TYPE      REASON          MESSAGE
Warning   Unschedulable   4/4 tasks in gang unschedulable: pod group is not ready, 4 Pending, 4 minAvailable;
                             Pending: 1 Unschedulable, 3 Schedulable.
                             Origin reason is gang-constrained-0-3: 0/4 nodes are unavailable:
                             1 Insufficient cpu, 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: },
                             2 node(s) had untolerated taint {test: blocked}.

When you free one of the tainted nodes, the entire gang launches together:

$ kubectl taint nodes llm-d-demo-worker2 test=blocked:NoSchedule-
$ kubectl get podgroups -n gang-demo
NAME                           STATUS    MINMEMBER   RUNNINGS
<pod-group-name>              Running   4           4

Takeaway: gang scheduling prevents the classic deadlock where one leader runs, waits forever for its workers, and hogs resources along the way.

Scenario 3 – Default Scheduler and the Sad Trombone

Manifest: manifests/examples/no-gang.yaml

• Same resource profile, but this time it’s a plain Deployment using the default scheduler.
• One lucky pod finds room and starts Running; the rest sit Pending.
• If you compare this output to Scenario 2, the difference is obvious: no PodGroup, no atomicity, and a partially launched workload that can’t make progress.

$ kubectl get pods -n gang-demo -l app=no-gang
NAME                            READY   STATUS    RESTARTS   AGE
no-gang-test-6686c8476b-28n8p   1/1     Running   0          5s
no-gang-test-6686c8476b-4jmdw   0/1     Pending   0          5s
no-gang-test-6686c8476b-9vnwv   0/1     Pending   0          5s
no-gang-test-6686c8476b-fq92q   0/1     Pending   0          5s

Takeaway: gang scheduling isn’t a luxury — it’s the difference between a healthy rollout and a half-deployed mess.

Try It Yourself

1. ./scripts/setup-cluster.sh
2. ./scripts/run-demo.sh (walks through the three scenarios interactively)
3. ./scripts/verify-gang-scheduling.sh (grabs PodGroup fields and events for receipts)
4. Optional cleanup: ./scripts/cleanup.sh

You can also poke through the live resources with kubectl get podgroups -n gang-demo or kubectl describe podgroup <pod-group-name> -n gang-demo to watch Volcano flip PodGroups from Inqueue to Running.

What to Remember

• Gang scheduling is all about all-or-nothing deployments; anything less invites deadlocks.
• LWS v0.7.0+ adds the tooling you need to pair with a gang scheduler without heavy lifting.
• Volcano acts as the enforcement engine, respecting minMember and minResources before letting pods bind.
• The moment you remove gang scheduling, the safety net goes away and partial rollouts come back.

Keep that mental model handy next time someone wonders why their distributed workload got itself stuck in Pending. Gang scheduling, done right, keeps the whole crew moving together.