Docker on Amazon EC2

Question

I am running several Docker containers on AWS (EC2, T2 medium) and I constantly run out of diskspace. It has something to do with the underlying filesystem. I run Docker on Ubuntu and it works fine (AUFS). The default AWS Linux versions are RHEL based and should use device mapper. This seems much more complicated, but to my surprise there is no decent manual how to use it (or I couldn't find it).

I can use the Ubuntu image on EC2, but I would think that it should be possible to run Docker on EC2 without to much issues? Or do I need to do something special to prevent it from running out of disk space?

I only use an NGINX and NodeJS image.

Answer 1

By default, docker will store it's data under /var/lib/docker. Is this path also on the root partition?

Also, by default when you create an EC2 instance the root volumes are usually small (around 4GB). It's highly likely that your root partition is also where your docker images are being stored and your root partition is too small.

So either, make your root partition bigger. Or, attach a new EBS volume, mount it somewhere sensible like /var/lib/docker and tell docker to use that new EBS volume for storing images.

If you want to change the directory where docker stores it's images away from the default /var/lib/docker then use the -g option when starting the docker daemon.

See: How do I change the docker image installation directory for more details on how you can change the default docker working path.

Answer 2

The Amazon Linux versions use device mapper as the Docker storage driver. Device mapper uses layered copy-on-write block devices to the containers. When no data has been written, then it hardly uses any disk spaces. As pages in the file system are being written, then it will allocate data from the device mapper and it will start using diskspace.

The default disk size for device mapper is 10GB, so when running 6 images you might require 60GB if all pages are being written to. It's unlikely that all pages are being written to, but the default EC2 instance only has 8GB which is not enough.

When a file is written then it is written to several blocks. If the blocks haven't been written yet, then it will be allocated from the device pool. When the file is removed, then the block is marked as unused in the filesystem, but the pool doesn't know about it. Only when fstrim is ran, then the filesystem releases the pages from the device manager.

NGINX maintains a cache on /tmp/cache and it's constantly written to. If the filesystem keeps using different blocks, then it will keep allocating blocks until all 10GB is mapped to the device manager. There are several solutions to this problem:

1. Use a device mapper device that is large enough to fit all the blocks.
2. Use a smaller base size (search for dm.basesize) to make sure the images cannot reach 10GB.
3. Switch to a distribution that uses AUFS storage.

You can also run fstrim regularly to make sure freed blocks are returned to the pool. This might be tricky, because during peak load it might not run often enough and the pool might still run out of disk space.