Set up GPU burst
Credentials for each burst location, the burst gateway, opting workloads in, and testing burst before production.
On this page
Burst attaches GPU nodes from other regions, other clouds or your own data center to a cluster whose home region has run out of GPUs. The cluster, its control plane and your tooling stay where they are. Set it up after the optimizer is running.
How burst nodes connect
- With burst enabled, the optimizer runs a burst gateway in your cluster, exposed through a load balancer you choose. Make it internal when your locations reach it over private connectivity.
- Each burst node opens an encrypted tunnel to the gateway. Kubelet registration, pod traffic and service traffic between the burst node and the cluster travel through it.
- Every burst node is tainted with
gpu.kubeon.io/burstand labelled with its location, so only workloads that opt in are scheduled there, and you can see where each pod runs.
1. Grant credentials for each location
The optimizer creates and removes burst nodes with credentials you grant for each location, scoped to the instances it creates. On AWS, a role like this lets it launch and terminate only instances carrying its burst tag:
AWS: burst role for eu-central-1
{ "Version": "2012-10-17", "Statement": [ { "Sid": "Discover", "Effect": "Allow", "Action": [ "ec2:DescribeInstances", "ec2:DescribeInstanceTypes", "ec2:DescribeInstanceTypeOfferings", "ec2:DescribeSpotPriceHistory", "ec2:DescribeImages", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups" ], "Resource": "*" }, { "Sid": "LaunchOnlyTaggedInstances", "Effect": "Allow", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:eu-central-1:111122223333:instance/*", "Condition": { "StringEquals": { "aws:RequestTag/kubeon.io/burst": "inference-burst-eu" } } }, { "Sid": "LaunchResources", "Effect": "Allow", "Action": "ec2:RunInstances", "Resource": [ "arn:aws:ec2:eu-central-1:111122223333:subnet/subnet-0b1c2d3e4f5a6b7c8", "arn:aws:ec2:eu-central-1:111122223333:security-group/sg-0a1b2c3d4e5f6a7b8", "arn:aws:ec2:eu-central-1:111122223333:network-interface/*", "arn:aws:ec2:eu-central-1:111122223333:volume/*", "arn:aws:ec2:eu-central-1::image/*" ] }, { "Sid": "TagAtLaunchOnly", "Effect": "Allow", "Action": "ec2:CreateTags", "Resource": "arn:aws:ec2:eu-central-1:111122223333:*/*", "Condition": { "StringEquals": { "ec2:CreateAction": "RunInstances" } } }, { "Sid": "TerminateOwnInstancesOnly", "Effect": "Allow", "Action": "ec2:TerminateInstances", "Resource": "arn:aws:ec2:eu-central-1:111122223333:instance/*", "Condition": { "StringEquals": { "aws:ResourceTag/kubeon.io/burst": "inference-burst-eu" } } }, { "Sid": "BurstNodeRole", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::111122223333:role/kubeon-burst-node" } ]}| Location | What to grant |
|---|---|
| AWS region or account | A role like the one above, with a trust policy for the optimizer's workload identity, and an instance profile kubeon-burst-node for the nodes |
| Azure region | A custom role on one resource group that can create, read and delete virtual machines and network interfaces there, assigned to the optimizer's managed identity |
| Google Cloud region | A custom role on one project that can create, get, list and delete instances and use one subnetwork, granted to the optimizer's service account |
| Your data center | Register idle GPU servers in KubeOn under the rate card's pool. They join through the same gateway with a join token, and return to their pool when released |
2. Enable the gateway and locations
Optimizer values
# Enable burst in the optimizer's valuesburst: enabled: true gateway: service: type: LoadBalancer annotations: # internal, if locations reach it privately service.beta.kubernetes.io/aws-load-balancer-scheme: internal locations: - name: aws/eu-central-1 credentials: awsRoleArn: arn:aws:iam::111122223333:role/kubeon-burst-eu-central-1 - name: gcp/europe-west4 credentials: gcpServiceAccount: kubeon-burst@acme-ml.iam.gserviceaccount.com - name: onprem/dc1 credentials: existingSecret: kubeon-burst-dc1 # join token for registered servers3. Opt workloads in
A burst-ready workload
# Opt a workload into burst: tolerate the burst taint.# Pods without this toleration never run on burst nodes.spec: template: spec: tolerations: - key: gpu.kubeon.io/burst operator: Exists effect: NoSchedule containers: - name: infer resources: limits: nvidia.com/gpu: 1Good candidates are stateless inference and batch jobs that can start anywhere their GPU fits. Keep stateful workloads and anything latency-critical on home capacity by leaving the toleration out.
4. Write the policy
Burst policy
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata: name: inference-burst-euspec: kind: Burst mode: Approve workloads: tolerate: gpu.kubeon.io/burst # only pods that opt in locations: # nothing outside this list, ever - aws/eu-central-1 - gcp/europe-west4 - onprem/dc1 limits: maxNodes: 4 maxGpuHour: 4.50 # USD, data transfer included returnHome: true # drain burst nodes when home capacity is backFor EU residency, list only EU regions and your own data centers. The optimizer never creates a node in a location missing from locations, and the audit log records the location of every burst node.
5. Test before production
- Start in Approve mode in a non-production cluster.
- Make a test workload pending, for example by requesting a GPU type the home region does not offer, with the toleration set.
- Approve the burst, check the node joins with its taint and location label, and that the pod reaches in-cluster services.
- Remove the test workload and check that the burst node drains and is removed.
- Compare the data transfer estimate shown before the burst with the transfer charges on the next bill.
Data transfer and latency
Traffic between locations is billed as data transfer and adds latency. KubeOn prices each location with an estimate of transfer back to the home region before a burst, but the bill is the final word.
Need help with this guide?Contact support