Troubleshooting

This section presents some common deployment errors and how to troubleshoot them.

Pod stuck in Pending state

A Pod can remain in Pending state for various reasons.

kubectl get po
NAME                                       READY   STATUS    RESTARTS   AGE
www                                        0/1     Pending   0          6s

To get more information about the root cause, run the following command (replacing the placeholders with the name of the Pod and the name of the Namespace in which it is running).

kubectl describe pod POD_NAME -n NAMESPACE

The Pod requires more resources than available

Let’s consider a Pod named www for which the describe command returns information similar to the following.

$ kubectl describe po www
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  110s  default-scheduler  0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.

This means the Pod’s specification requests more resources than currently available. There are several cases:

• the Pod’s container actually needs this amount of resources

  In this case, adding a Node with more resources could fix the issue

• the requests were not configured currently and are higher than what is needed

  In this case, decreasing the resources requests (for memory and/or CPU), so it fits the application needs, should be explored. The Pod specification below shows these properties for a simple nginx container.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: www
  name: www
spec:
  containers:
  - image: nginx:1.24
    name: www
    resources:
      requests:
        memory: 4G     # memory and cpu requests are taken into account during the scheduling phase
        cpu: 2         # decreasing the amount of mempry and cpu may fix the issue in some cases

The Pod has a placement constraint that cannot be satisfied

Some Pods need to be scheduled on a specific type of Nodes; for instance, an ML/AI workload usually needs to run on a Node with a GPU. These schedule constraints can be defined in the Pod specification as in the example below.

apiVersion: v1
kind: Pod
metadata:
  name: ml
spec:
  containers:
  - image: myMLApp:1.4.2
    name: ml
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values:
            - gpua5000     # This Pod needs to be deployed on a node with 
            - gpu3080ti    # a 3080TI or a A5000 GPU

When run on a cluster which does not have this type of Node, this Pod will remain in Pending state, the output of the describe command being similar to the one below.

$ kubectl describe pod ml
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  11s   default-scheduler  0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

To fix this issue, we could either:

• modify the Pod specification referencing a GPU instance type existing in the cluster
• add a new Node into the cluster, this one of the type defined in the Pod specification

The Pod needs storage that cannot be created

Some Pods need persistent storage to store data; for instance, a database needs to persist data on a real storage, not in the container’s file system. This is done through the usage of a PersistentVolumeClaim resource, which is referenced in the Pod specification.

The example below defines a Pod running MongoDB and persisting its data via a PVC named mongo-data.

  apiVersion: v1
  kind: Pod
  metadata:
    name: mongo
  spec:
    containers:
    - image: mongo:6
      name: mongo
    volumes:
    - name: data
      persistentVolumeClaim:
        claimName: mongo-data

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: mongo-data
  spec:
    storageClassName: manual
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi

• Non-existing StorageClass

There may be situation where the describe command returns an error message similar to the following one. This indicates that no PersistentVolume could be created and associated to the PVC.

...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  43m   default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

Using the describe command on the PVC may provide additional information. In the example below, the error is due to an invalid name for the StorageClass.

$ kubectl describe pvc mongo-data
...
Events:
  Type     Reason              Age                     From                         Message
  ----     ------              ----                    ----                         -------
  Warning  ProvisioningFailed  3m22s (x481 over 166m)  persistentvolume-controller  storageclass.storage.k8s.io "manual" not found

We can change the PVC definition, using the name of an existing StorageClass (local-path in this example).

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: mongo-data
    spec:
      storageClassName: local-path
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

After recreating the PVC using the updated specification, the PVC should be bound to a PV, and the Pod should transition to the Running state.

• There may be other reasons which prevent the underlying PV to be created such as capacity limits of the underlying storage