GPU policy reference
Every field of a GpuPolicy, for Sharing, Autoscaling, ScaleToZero and Burst policies, with examples.
A policy tells the optimizer what it may change, where, how far and when. Anything a policy does not name is out of the optimizer's reach. Create policies in KubeOn or with the API; they are synced into each cluster as GpuPolicy resources.
Fields every policy has
| Field | Meaning |
|---|---|
kind | Sharing, Autoscaling, ScaleToZero or Burst |
mode | Recommend, Approve or Automatic. Defaults to the optimizer's defaultMode, which is Recommend |
nodePools, nodeSelector or namespaces | The only nodes or namespaces the policy can act on |
window | When disruptive steps may run, such as drains and repartitions, in the form Mon-Fri 19:00-23:00 UTC |
limits | Hard limits: minNodes, maxNodes, maxMonthlyCost at your billed rates, and for burst maxGpuHour |
Sharing
Sharing policy
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata: name: dc1-migspec: kind: Sharing mode: Approve # each repartition waits for a reviewer nodeSelector: node.kubernetes.io/pool: gpu mig: allowedProfiles: [1g.5gb, 2g.10gb, 3g.20gb] headroom: 20% # above peak GPU memory timeSlicing: namespaces: [ml-dev] # non-production only maxReplicas: 4 window: "Sat 02:00-06:00 UTC"| Field | Meaning |
|---|---|
mig.allowedProfiles | MIG profiles the optimizer may use. Profiles must be valid for the GPU model |
mig.headroom | Room kept above each workload's peak GPU memory when choosing a profile |
timeSlicing.namespaces | The only namespaces whose workloads may share a time-sliced GPU |
timeSlicing.maxReplicas | The most pods that may share one GPU |
Autoscaling
Autoscaling policy
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata: name: ml-inference-autoscalingspec: kind: Autoscaling mode: Automatic # Recommend | Approve | Automatic nodePools: [ml-inference] limits: minNodes: 1 maxNodes: 6 maxMonthlyCost: 40000 # USD at billed rates capacity: order: [spot, on-demand] # spot first, on-demand when spot is gone packing: enabled: true window: "Mon-Fri 19:00-23:00 UTC" respectPodDisruptionBudgets: true| Field | Meaning |
|---|---|
capacity.order | Capacity types to try, in order: spot, then on-demand. Leave out on-demand to stay on spot only |
packing.enabled | Whether the optimizer may drain and remove nodes once their pods fit elsewhere |
packing.window | When packing may run |
packing.respectPodDisruptionBudgets | Always true: evictions go through the Eviction API |
ScaleToZero
Scale-to-zero policy
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata: name: notebooks-scale-to-zerospec: kind: ScaleToZero mode: Automatic namespaces: [ml-notebooks] # only namespaces you list idle: gpuUtilizationBelow: 2% # DCGM; 2% is the default for: 30m wake: schedule: "Mon-Fri 07:00 UTC" removeEmptyNodes: true| Field | Meaning |
|---|---|
idle.gpuUtilizationBelow | The idle threshold, from DCGM_FI_DEV_GPU_UTIL. Defaults to 2% |
idle.for | How long a workload must stay under the threshold before it is scaled to zero |
wake.schedule | When scaled-down workloads return. They also return as soon as someone scales them up |
removeEmptyNodes | Whether GPU nodes left empty are removed |
Burst
Burst policy
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata: name: inference-burst-euspec: kind: Burst mode: Approve workloads: tolerate: gpu.kubeon.io/burst # only pods that opt in locations: # nothing outside this list, ever - aws/eu-central-1 - gcp/europe-west4 - onprem/dc1 limits: maxNodes: 4 maxGpuHour: 4.50 # USD, data transfer included returnHome: true # drain burst nodes when home capacity is back| Field | Meaning |
|---|---|
workloads.tolerate | The taint every burst node carries. Only pods that tolerate it can run there |
locations | The only places burst nodes may be created. A location not on this list is never used |
limits.maxNodes, limits.maxGpuHour | The most burst nodes at once, and the highest GPU-hour price including estimated data transfer |
returnHome | Drain pods back and remove burst nodes once home capacity is available |
Status
Each GpuPolicy reports its last action, the actions waiting for approval and any limit that stopped it, in status. The same information, with who approved what, is in KubeOn's audit log, where each action that can be undone has a revert.
Need help with this guide?Contact support