Orient Me: Rethink your Docker storage driver configuration

connections cnx

I did some research around Docker storage drivers, CfC and Orient Me the last days because I had some issues in different Orient Me installations. Finally, I came to the conclusion that the Docker storage driver configuration which is installed by the Orient Me setup might need to be customized for a production environment.

Why?

An Orient Me installation uses the overlay storage driver which is the default configuration when installing Docker from the official repository using yum. The overlay storage driver is default since Docker 1.13.1 (if the kernel supports it). The support for overlay was also added with RHEL 7.3 which is a requirement for running Orient Me. But, overlay with REHL 7.3 is only supported with XFS as the backing filesystem. This might be the first hook for some installations out there. But more important: If you check the Docker documentation you will find the information that devicemapper is the recommended and tested/supported storage driver which should be used with RHEL/CentOS and Docker CE. The same is mentioned for Docker EE. While researching this I got the same response from the IBM Cloud private team. They mentioned that devicemapper storage driver should be used with CfC/ICp on RHEL or CentOS.

After changing the storage driver to devicemapper on one of my test environment two of my known issues were fixed:

  • Issues related to CFC indices cleaner cronjob scheduling (Stoeps blogged about it here)
  • Increased CPU usage of the Docker daemon (dockerd) process

How to?

There is no migration path available which allows you to move your containers and images from the overlay to the devicemapper storage driver. Therefore the fastest solution is to reinstall Orient Me services and CfC. To uninstall Orient Me please use the deployCfC.sh which will also uninstall Docker itself.

  1. Reinstall Docker-CE 17.03 (or the supported EE version) and all dependencies which are required to run CfC 1.1:
    yum install -y yum-utils device-mapper-persistent-data lvm2
    yum-config-manager —add-repo https://download.docker.com/linux/centos/docker-ce.repo
    yum-config-manager --disable docker-ce-edge
    yum-config-manager --enable docker-ce-stable
    yum makecache fast
    yum -y --setopt=obsoletes=0 install docker-ce-17.03.2.ce-1.el7.centos
    systemctl enable docker.service
    systemctl start docker
  2. Check if Docker is installed and running:
    systemctl status docker
    docker info
    docker run hello-world
  3. Prepare Docker:
    1. Stop the docker daemon using systemctl stop docker
    2. Backup and move your /var/lib/docker directory
  4. Configure the volume:
    Because of performance reasons do not use loop-lvm in production. I will describe how to configure direct-lvm. Therefore you need to add or pick a physical device which we will use as thinpool to store your containers and images. In my case, it will be /dev/sdb.

    pvcreate /dev/sdb
    vgcreate docker /dev/sdb
    lvcreate --wipesignatures y -n thinpool docker -l 95%VG
    lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
    lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta

    You can configure to extend the logical volumes automatically if needed. You will find more information here.

  5. Reconfigure Docker:
    With Docker 17.06 there is simplified configuration available. In our case, we need to configure direct-lvm manually. Create or edit /etc/docker/daemon.json with following content:

    {
    "storage-driver": "devicemapper",
    "storage-opts": [
    "dm.thinpooldev=/dev/mapper/docker-thinpool",
    "dm.use_deferred_removal=true"
    ]
    }

    Please add dm.use_deferred_deletion=true in case you are using a 4.x kernel.

  6. Start Docker and verify your configuration:
    systemctl start docker
    docker info
    docker run hello-world

    Please verify following docker info output:

    Storage Driver: devicemapper
    Pool Name: docker-thinpool
    Pool Blocksize: 524.3 kB
    Base Device Size: 10.74 GB
    Data file:
    Metadata file:
  7. Reinstall CfC and Orient Me using deployCfC.sh:
    You need to include –skip_docker_deployment to be sure that the installer will not reinstall Docker again.

Update 17/09/28

IBM will officially support the devicemapper storage driver with future Orient Me versions as mentioned here.

Leave a Reply

Your email address will not be published. Required fields are marked *