GPU sharing
Workloads that use a slice of a GPU, given a slice
KubeOn finds workloads whose GPU compute and memory fit in part of a card, and proposes how to share it: MIG partitions where the GPU supports them and isolation matters, time-slicing for development. Each proposal carries the configuration, the trade-off and the GPUs it frees.
GPU sharing workflow
Who does what
From a whole GPU to the slice a workload needs
Follow the inputs, decision and output for this workflow.
- KubeOn, automated
- Your decision
- Your systems and people
- Step 1, Measure, by Agent + Prometheus:
Peak GPU compute and memory per workload over its history, from DCGM metrics.
- Step 2, Match, by KubeOn hub:
Fit each under-used workload to a MIG profile on a supported GPU, or to time-slicing for development.
- Step 3, Approve, by Your reviewers:
Review the isolation trade-off, the risk level and the rollout steps for the node pool.
- Step 4, Apply, by Your pipeline:
Your pipeline applies the GPU Operator configuration and the pods' new GPU resources.
KubeOn reads and recommends. Your team and pipeline change the cluster.
Read the workflowWho does what
From a whole GPU to the slice a workload needs
Follow the inputs, decision and output for this workflow.
- 01
Measure
Peak GPU compute and memory per workload over its history, from DCGM metrics.
Agent + Prometheus
- 02
Match
Fit each under-used workload to a MIG profile on a supported GPU, or to time-slicing for development.
KubeOn hub
- 03
Approve
Review the isolation trade-off, the risk level and the rollout steps for the node pool.
Your reviewers
- 04
Apply
Your pipeline applies the GPU Operator configuration and the pods' new GPU resources.
Your pipeline
KubeOn reads and recommends. Your team and pipeline change the cluster.
Read the workflowThe problem
One small service, one whole GPU
Whole-device allocation
By default the device plugin gives each GPU to one container, however little of it the container uses.
Sharing has trade-offs
MIG isolates memory and faults but has fixed sizes; time-slicing is flexible but isolates nothing.
Not every GPU can partition
MIG runs on data center GPUs such as A100 and H100, not on L40S, L4 or A10G.
Changes touch the node
Applying a MIG profile reconfigures the GPU, so the node's GPU pods have to move first.
Sharing candidates
Workloads whose peak fits in part of a card
From DCGM history, KubeOn takes each GPU workload's peak GPU memory and compute. Workloads that hold a whole GPU but whose peak memory fits a smaller slice, with headroom, become sharing candidates, grouped by the node pool that would change.
- Peak GPU memory, not the average, decides the slice
- Grouped by node pool, so one change covers several workloads
- Production and non-production treated differently
Multi-Instance GPU
MIG where the GPU supports it and isolation matters
On GPUs with MIG, such as A100 and H100, a card is split into isolated instances with their own memory and compute. KubeOn picks the smallest valid profile that holds each workload's peak memory, packs workloads onto as few GPUs as possible, and writes the node label and pod resource changes for the GPU Operator.
- Only profiles valid for that GPU model
- Hardware isolation of memory and faults
- Rollout steps include draining the node's GPU pods
# One dc1 A100 40GB node, partitioned by the GPU Operator MIG managerkubectl label node dc1-gpu-07 nvidia.com/mig.config=all-2g.10gb --overwrite# embed-svc, classify-svc and ocr-svc request a slice, not a GPUresources: limits: nvidia.com/mig-2g.10gb: 1 # was nvidia.com/gpu: 1Time-slicing
Time-slicing for notebooks, development and small batch
Time-slicing lets several pods take turns on one GPU on any NVIDIA card, including those without MIG. It has no memory or fault isolation, so KubeOn proposes it for non-production namespaces and marks the risk on every proposal.
- Replicas per GPU from peak memory, so they fit together
- Non-production by default
- GPU Operator config map and node label written out
# dev-use1 / ml-dev: four notebook pods per A10G, non-production onlyapiVersion: v1kind: ConfigMapmetadata: name: time-slicing-config namespace: gpu-operatordata: a10g: |- version: v1 sharing: timeSlicing: resources: - name: nvidia.com/gpu replicas: 4# then: kubectl label node <node> nvidia.com/device-plugin.config=a10gChoosing a method
MIG, time-slicing or neither
| MIG | Time-slicing | Neither | |
|---|---|---|---|
| GPUs | Data center GPUs with MIG, such as A100 and H100 | Any NVIDIA GPU with the device plugin | Any |
| Isolation | Memory, compute and faults isolated in hardware | None between the pods sharing a GPU | Whole GPU per container |
| Sizes | Fixed profiles per GPU model | Any number of replicas | Whole GPUs |
| Applying it | Node reconfigured; GPU pods drained first | Device plugin config; pods restarted | No change |
| KubeOn proposes it for | Small production inference that needs isolation | Notebooks, development and small batch | Workloads that keep a GPU busy |
A training job running at 90% on 8 GPUs needs nothing. A service holding one GPU at 6% is a sharing or hosting candidate. The measurement comes first.
Capabilities
Sharing your reviewers can approve
MIG profiles
The smallest valid profile for each workload's peak memory.
Time-slicing
Replicas per GPU for development and notebooks.
Peak-based fit
DCGM memory and compute peaks, with headroom.
Isolation stated
Each proposal says what is and is not isolated.
GPUs freed
Whole GPUs released, priced at the node's rate.
Reviewable changes
Node labels, config maps and pod resources as a patch.
FAQ
GPU sharing, answered
No. KubeOn proposes the profile and writes the node label and pod changes. Your team approves the proposal and your pipeline applies it, usually through the NVIDIA GPU Operator's MIG manager and device plugin.
Related
Evaluate GPU sharing with your team.
Bring your platform and FinOps priorities. We will walk through the data, controls and evidence for your evaluation.