what is the difference between used & hard quota for memory in k8s?

Background

Resource Quota is limit of specific resource which is namespace wide.

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace.

As I mention in comments it's very similar to Requests and Limits which are used in Pods. Request is value which is requested by pod on the beginning and limit is maximum value which pod can use. In Quota, hard value is like limit, it cannot have more than this value. Used is value of current use of this particular resource. Quota can be used to limit number of jobs, pods, services, cpu or memory.

Tests

To list all your quotas, you can use command $ kubectl get quota -A. If you miss -A or --all-namespaces you will get only quota from default namespace.

I've tested this on GKE cluster and Minikube.

As default, GKE cluster have a few Quotas. You can have more than 1 Quota in namespace. In below example you have 2 quotas for kube-system namespace.

$ kubectl get quota -A
NAMESPACE         NAME                  AGE   REQUEST                      LIMIT
default           gke-resource-quotas   11m   count/ingresses.extensions: 0/100, count/ingresses.networking.k8s.io: 0/100, count/jobs.batch: 0/5k, pods: 0/1500, services: 1/500
kube-node-lease   gke-resource-quotas   11m   count/ingresses.extensions: 0/100, count/ingresses.networking.k8s.io: 0/100, count/jobs.batch: 0/5k, pods: 0/1500, services: 0/500
kube-public       gke-resource-quotas   11m   count/ingresses.extensions: 0/100, count/ingresses.networking.k8s.io: 0/100, count/jobs.batch: 0/5k, pods: 0/1500, services: 0/500
kube-system       gcp-critical-pods     18m   pods: 13/1G 
kube-system       gke-resource-quotas   11m   count/ingresses.extensions: 0/100, count/ingresses.networking.k8s.io: 0/100, count/jobs.batch: 0/5k, pods: 15/1500, services: 3/500

But Minikube don't have any at the beginning. To enable Resource Quota, please check this docs

You can describe Quota to check which resources are going to be limited.

$ kubectl describe quota gke-resource-quotas -n kube-system
Name:                              gke-resource-quotas
Namespace:                         kube-system
Resource                           Used  Hard
--------                           ----  ----
count/ingresses.extensions         0     100
count/ingresses.networking.k8s.io  0     100
count/jobs.batch                   0     5k
pods                               15    1500
services                           3     500

Based on the above output:

In my current GKE cluster I have 15 pods running (Used) but maximum number of pods I can have in this namespace is 1500. I won't be able to deploy in this namespace 1501 pods.

Create namespace

$ kubectl create ns test

Creating Quota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: test
  namespace: test
spec:
  hard:
    requests.memory: "100Mi"  
    limits.memory: "200Mi"

resourcequota/test created

$ kubectl describe quota -n test
Name:            test
Namespace:       test
Resource         Used  Hard
--------         ----  ----
limits.memory    0     200Mi
requests.memory  0     100Mi

In above example I don't have any pods whih are using memory.

Scenario

2 pods with 50Mi requests and 100Mi Limit each.

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-1
  namespace: test
spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      requests:
        memory: "50Mi"
      limits:
        memory: "100Mi"

After that, Used value of Quota has been chagnged.

$ kubectl describe quota -n test
Name:            test
Namespace:       test
Resource         Used   Hard
--------         ----   ----
limits.memory    100Mi  200Mi
requests.memory  50Mi   100Mi

If I would create second pod with the same values but with name nginx-2, Kubernetes allows me to do it as Hard values won't be exceeded. Used and Hard values will be the same

$ kubectl describe quota -n test
Name:            test
Namespace:       test
Resource         Used   Hard
--------         ----   ----
limits.memory    200Mi  200Mi
requests.memory  100Mi  100Mi

However, I won't be able to create nginx-3 as it would exceeded Hard value. You will receive error:

Error from server (Forbidden): error when creating "pod.yaml": pods "nginx-3" is forbidden: exceeded quota: test, requested: limits.memory=100Mi,requests.memory=50Mi, used: limits.memory=200Mi,requests.memory=100Mi, limited: limits.memory=200Mi,requests.memory=100Mi

In above example resourcequota/test set limitations on test namespace.

• limits.memory is referring to pod's spec.containers.resources.limits.memory.
• requests.memory is referring to pod's spec.containers.resources.requests.memory.

Created quota won't allow to create pods if limits and requests will exceed Hard value in the quota.

In $ kubectl describe quota test -n test, Usage value is sum of all pod's limits in this namespace (currently in use). Hard is maximum value which cannot be exceeded.

Conclusion

Used value of Quota is number of resources which are already using in this namespace. It can be number of pods or amount of memory or cpu. Hard value of Quota is maximum number of pods or maximum threshold for memory to use. If it will exceed Hard value, particular resource won't be created.

Useful links

• How to use Kubernetes resource quotas
• Limit Ranges

Please let me know if you still have some questions.