271

A pod in my Kubernetes cluster is stuck on "ContainerCreating" after running a create. How do I see logs for this operation in order to diagnose why it is stuck? kubectl logs doesn't seem to work since the container needs to be in a non-pending state.

Improve this question
2
  • kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/… is the documentation on the possible phases. Unfortunately it doesn't include ContainerCreating... Commented Jul 19, 2017 at 21:46
  • 4
    Usually when I get this issue it's because the appropriate secrets aren't created - kubectl describe pods *pod_name* will reveal if this is the cause - look at the 'events' listed at the bottom of the output. Tip - to get the pod_name use kubectl get pods, and copy the name of the pod you want to inspect. Commented Oct 7, 2020 at 23:16

6 Answers 6

Reset to default
322

kubectl describe pods will list some (probably most but not all) of the events associated with the pod, including pulling of images, starting of containers.

Improve this answer
5
  • 8
    what if the container stuck at ContainerCreating without any events? for me the events is shown as "No events." Commented May 26, 2016 at 2:30
  • 1
    Some events seem to take a while to show up. For example a timeout attempting to mount a disk for me takes about 2 minutes before it shows up as an event. Commented Jul 28, 2016 at 18:49
  • 20
    It happens when you are using secrets and they are not found (like a typo in the yaml or you forgot to create it before). For the almost all other possible errors it gets CrashLoopback or Error states but with secrets it just gets stuck in ContainerCreating, if you describe the pod then you'll see at the very end a message saying the secret was not found, but it barely says nothing about the problem. Commented Oct 31, 2016 at 20:53
  • Yeah usually you don't have any events before he starts doing something. Commented May 10, 2017 at 14:44
  • When the image being pulled from docker hub is relatively large, The pod status can be stuck at ContainerCreating for a while before changing state to Running. In my case, i just waited for a little longer and the state changed to running Commented Sep 7, 2021 at 8:31
70

More info could be provided in the events.

kubectl get events --all-namespaces  --sort-by='.metadata.creationTimestamp'

However do note that sorting events might not work correctly due to this bug: https://github.com/kubernetes/kubernetes/issues/29838

Alternatively:

As of Kubernetes 1.18 all new objects have metadata for server-side apply, which gives us a new way to sort events:

kubectl get events --sort-by=".metadata.managedFields[0].time"

From: https://github.com/kubernetes/kubernetes/issues/29838#issuecomment-789660546

In my case I had an event relating to a pod:

default       13s         Warning   FailedMount               Pod          Unable to mount volumes for pod "restore-db-123-1-5f24s_default(9b7df264-2976-11ea-bb8f-42010a9a002c)": timeout expired waiting for volumes to attach or mount for pod "default"/"restore-db-123-1-5f24s". list of unmounted volumes=[nfsv]. list of unattached volumes=[nfsv default-token-hxrng]
Improve this answer
3
  • Thank you for this! I was trying to use container logs using the GKE-provided queries, but I suspect my filters were too tight. This command helped me isolate what exactly was happening. (I was forgetting to build the configmap, derp.) Commented Oct 9, 2020 at 1:26
  • 1
    Nice tip on --sort-by :) Commented Jul 6, 2021 at 20:16
  • 1
    Much better than accepted answer for my case. Actually showed that I ran out of disk space helping me pinpoint the issue real quick. Thanks!! Commented Jan 17, 2022 at 8:46
7

In my case, docker's access to internet was blocked. It was solved using a proxy (using sandylss's comment):

  1. minikube stop
  2. minikube delete
  3. export http_proxy=http://user:pass@ip:port
  4. export https_proxy=http://user:pass@ip:port
  5. export no_proxy=192.168.99.0/24
  6. minikube start --logtostderr --v=0 --bootstrapper=localkube --vm-driver hyperv 
  --hyperv-virtual-switch "Primary Virtual Switch" --docker-env HTTP_PROXY=$http_proxy \
  --docker-env HTTPS_PROXY=$https_proxy --docker-env NO_PROXY=$no_proxy
  7. export no_proxy=$no_proxy,$(minikube ip)
  8. export NO_PROXY=$no_proxy,$(minikube ip)

Then, to check if docker has access to internet, run:

$ docker pull tutum/hello-world

in the cluster (connect to the cluster using minikube ssh); stop the process if it starts downloading.

My second problem was slow internet connection. Since the required docker images are on the order of 100MB, both docker containers and Kubernetes pods remained in \pause and ContainerCreating states for 30 minutes.

To check if docker is downloading the images, run:

$ ls -l /var/lib/docker/tmp

in the cluster, which shows the temporary image file[s] that are being downloaded, empty otherwise.

If you are developing in minikube and using VPN, docker can use your VPN via fiddler. That is, docker will be connected to fiddler's ip:port, and fiddler is connected to the VPN. Otherwise, VPN is not shared between your host and minikube VM.

Improve this answer
1
  • Got bit by this bug today. Still not sure what caused it though. Things were working fine one minute and the next, this issue cropped up. Thank you for the fix. It worked for me. Commented Oct 31, 2018 at 6:45
6

In my case, a pod was stuck at 'ContainerCreating' because a docker image pull was hung (some layers were downloaded, some were stuck at "downloading").

$ kubectl get events --all-namespaces  --sort-by='.metadata.creationTimestamp'

showed an event "Pulling image"

Tried to pull that image using docker image pull... and saw that it was hanging.

It turned out that there is a bug in concurrent pulls of layers. Changing docker config to limit concurrency solved the problem.

Added this to docker config (on windows, docker-desktop UI, settings, Docker Engine) to limit concurrency:

  "max-concurrent-downloads": 1,
  "max-concurrent-uploads": 1
Improve this answer
3

The one time I hit this was because my resource declarations were accidentally very very small.

resources: limits: cpu: 1000m memory: 1024M requests: cpu: 1000m memory: 1024M

vs

resources: limits: cpu: 1000m memory: 1024m requests: cpu: 1000m memory: 1024m

capitalizing that m makes a very large difference in resource use. I was stuck on ContainerCreating because I had not given enough memory to my container.

Improve this answer
0

I've been there! I had a similar issue where my StatefulSet pod was stuck in the ContainerCreating status. After spending about three hours troubleshooting, I decided to cordon the node where the NFS server was located and redeploy it to another node.

This did the trick for me! My service started running smoothly once the volume was correctly bound to the new NFS server location.

Hope this helps!

Improve this answer
1
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. Commented Jul 10, 2024 at 6:56

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.