About:
This article contains Kubernetes resource quota examples for limiting GPU usage per namespace.
For further information on setting resources quotas against namespaces in Kubernetes: https://kubernetes.io/docs/concepts/policy/resource-quotas/#viewing-and-setting-quotas
Applying a resource quota
In this example, we have the user “john” who has a Kubernetes namespace called “john-restricted”.
In order to define limits for namespaces, you will need to define resource quotas against that specific namespace.
In this case, Nvidia Bright does not control the Kubernetes resource quotas, and the commands in order to define the resource quota are the same as for any standard Kubernetes setup. For example, define a resource quota that has a resource limit of one “nvidia.com/gpu” resource, then assign this to the john-restricted namespace as follows:
# cat compute-resources.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-resources
spec:
hard:
requests.nvidia.com/gpu: 1
Apply the compute-resources.yaml to the john-restricted namespace:
# kubectl create -f ./compute-resources.yaml --namespace=john-restricted resourcequota/compute-resources created
If the user john attempts to request more than 1 GPU in the restricted-john namespace, they will encounter the error:
"Error from server (Forbidden): error when creating "gpu-test-pod.yaml": pods "gpu-test-pod" is forbidden: exceeded quota: compute-resources, requested: requests.nvidia.com/gpu=2, used: requests.nvidia.com/gpu=0, limited: requests.nvidia.com/gpu=1