1. Home
  2. Containers
  3. Kubernetes: Limit GPU resource usage for a namespace

Kubernetes: Limit GPU resource usage for a namespace


This article contains Kubernetes resource quota examples for limiting GPU usage per namespace.
For further information on setting resources quotas against namespaces in Kubernetes: https://kubernetes.io/docs/concepts/policy/resource-quotas/#viewing-and-setting-quotas

Applying a resource quota

In this example, we have the user “john” who has a Kubernetes namespace called “john-restricted”.

In order to define limits for namespaces, you will need to define resource quotas against that specific namespace.
In this case, Nvidia Bright does not control the Kubernetes resource quotas, and the commands in order to define the resource quota are the same as for any standard Kubernetes setup. For example, define a resource quota that has a resource limit of one “nvidia.com/gpu” resource, then assign this to the john-restricted namespace as follows:

# cat compute-resources.yaml
apiVersion: v1
kind: ResourceQuota
  name: compute-resources
    requests.nvidia.com/gpu: 1

Apply the compute-resources.yaml to the john-restricted namespace:

# kubectl create -f ./compute-resources.yaml --namespace=john-restricted
resourcequota/compute-resources created

If the user john attempts to request more than 1 GPU in the restricted-john namespace, they will encounter the error:

"Error from server (Forbidden): error when creating "gpu-test-pod.yaml": pods "gpu-test-pod" is forbidden: exceeded quota: compute-resources, requested: requests.nvidia.com/gpu=2, used: requests.nvidia.com/gpu=0, limited: requests.nvidia.com/gpu=1
Updated on August 14, 2023

Related Articles

Leave a Comment