Aah yes, that age old problem, "it works on my machine". Docker has already proven itself to be an incredible tool to have in any developer or system administrator's toolbox, and while it does solve many problems, some still linger. In this case I'm talking about issues with cross-platform filesystem permissions for mounted volumes.

So what's the problem? What isn't working on other people's machines? Perhaps you're one of the lucky folks that have chosen "the right platform" and haven't experienced any issues. If you're not one of those lucky people, then you may have experienced times when containers aren't able to write to your host filesystem, or maybe it can, but then you've accidentally written the files as root to your host machine (maybe that's fine in some circumstances, but I'd prefer all my files to be owned by my user on the host).

This is a problem that is particularly frustrating for developers when working with Docker locally. Docker makes it very quick to get a project's specific environment up and running if you've got it configured correctly, but if you can't write files to the host, or can't edit files that a container has written on your host in your editor then it's going to become a pain point in your environment expeditiously.

There are a few solutions to this problem that I've seen, and I've tried to implement each of them and see how they go. At Byng we use Docker now for most of our local development environments, but we also allow developers to choose how they set up their machines, and choose what OS they use. This means that these local development environments must behave consistently regardless of what the environment of the host is. I'm going to outline the best solution I've found so far given this restriction.

The Requirements

Given the main problems I've seen with other solutions, here are some basic requirements that this solution aims to meet:

The Solution

This solution is one that quite a few official images use now. That being the case, it still isn't without it's own problems, but there's been nothing yet that's been a deal breaker for Byng. More on that later.

We're going to take advantage of Docker's ENTRYPOINT instruction. The ENTRYPOINT of a container is what the CMD is run via. By default the ENTRYPOINT of a container is /bin/sh -c. Your CMD is passed as a parameter to the ENTRYPOINT. So, if you were to leave the default ENTRYPOINT in, and had the command echo 'Hello, World!', then Docker is actually running this:

$ /bin/sh -c "echo 'Hello, World!'"

We can take advantage of this, and change the ENTRYPOINT so that it's a shell script. This allows us to always execute a script when $ docker run is used (you may be able to see where this is heading).

The script that we'll use as our ENTRYPOINT will do a few things:

  1. It'll check if a user with a given name exists, and if not, it'll create it with a pre-defined UID, GID, and home directory. These will be set with environment variables with sane defaults.
  2. It'll make sure the home directory of the user that was configured has been created, and will also ensure that it is owned by the correct user in the container.
  3. It'll create a working directory for code to be mounted into that's owned by the correct user in the container.
  4. Finally, it'll run the container's CMD. This may be run as root, or we may switch to the newly created user in the container before running the CMD. This will depend on what you're running. One example of where it may easier to start as root is with something like PHP-FPM. Here you configure a user for it to run as, and then PHP-FPM spawns workers as that user.

We're going to build on an Alpine 3.4 image, and we're going to use gosu. We'll be testing this image out with whoami. We'll make 3 files, a Dockerfile to build the image, a docker-entrypoint.sh shell script to serve as the ENTRYPOINT of the image, and a docker-cmd.sh shell script that will be our CMD. As a result, by default we'll be running the following when the image is run:

$ /docker-entrypoint.sh /docker-cmd.sh

Here is our Dockerfile:

# Dockerfile
FROM alpine:3.4  
MAINTAINER Elliot Wright <elliot@byng.co>

COPY ./provisioning/docker-entrypoint.sh /docker-entrypoint.sh  
COPY ./provisioning/docker-cmd.sh /docker-cmd.sh

ENV APP_UID 1000  
ENV APP_GID 1000  
ENV APP_HOME /home/app  
ENV APP_WORKDIR /opt/app  
ENV GOSU_VERSION 1.9

RUN set -x \  
    && apk add --no-cache --virtual .gosu-deps \
        dpkg \
        gnupg \
        openssl \
    && dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
    && wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch" \
    && wget -O /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc" \
    && export GNUPGHOME="$(mktemp -d)" \
    && gpg --keyserver ha.pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
    && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
    && rm -r "$GNUPGHOME" /usr/local/bin/gosu.asc \
    && chmod +x /usr/local/bin/gosu \
    && gosu nobody true \
    && apk del .gosu-deps
    && chmod +x /docker-entrypoint.sh \
    && chmod +x /docker-cmd.sh

ENTRYPOINT ["/docker-entrypoint.sh"]

CMD ["/docker-cmd.sh"]  

This Dockerfile copies our ENTRYPOINT and CMD scripts into the image, sets a few defaults for some environment variables we'll use to configure the image at runtime, and provides the version of gosu that we'll use when building. It then proceeds to install gosu, clean up after itself, and make our ENTRYPOINT and CMD scripts executable. As an aside, I've attempted to keep the size of the image reasonably small, as you always should when creating Docker images!

Next up we have our ENTRYPOINT script:

# provisioning/docker-entrypoint.sh
#!/bin/sh

USER=$(getent passwd app 2>&1)  
GROUP=$(getent group app 2>&1)

set -e

echo "==> Attempting to create 'app' user."  
if [ ${#USER} == 0 ] && [ ${#GROUP} == 0 ]; then  
    echo "    Creating 'app' user with UID:GID of ${APP_UID}:${APP_GID}"
    addgroup -g ${APP_GID} app
    adduser -u ${APP_UID} -G app -h ${APP_HOME} -HD app
else  
    echo "    User 'app' already exists."
fi

echo "==> Attempting to create home directory at '${APP_HOME}'."  
if [ ! -d "${APP_HOME}" ]; then  
    mkdir -p "${APP_HOME}"
    chown ${APP_UID}:${APP_GID} "${APP_HOME}"
    echo "    Done!"
else  
    echo "    Already exists!"
fi

echo "==> Attempting to create working directory at '${APP_WORKDIR}'."  
if [ ! -d "${APP_WORKDIR}" ]; then  
    mkdir -p "${APP_WORKDIR}"
    chown ${APP_UID}:${APP_GID} "${APP_WORKDIR}"
    echo "    Done!"
else  
    echo "    Already exists!"
fi

cd "${APP_WORKDIR}"

echo "==> Running CMD as 'app'..."  
exec gosu app "$@"  

This file is taking care of creating a user with our pre-configured UID, GID, and home directory. It also sets up a working directory that we will move into. The defaults for all of these values are specified in the Dockerfile, and are set to some fairly sensible values to make it useful out of the box. Each step of the way it checks if the user, home directory, and working directory already exists. This can be the case if you're simply restart a container (or in the case of the directories, if you are volume mounting them into the container).

Finally, we need our CMD script. This is only going to be for testing, so isn't going to be doing anything complex, we just need to see that we've met our requirements:

# provisioning/docker-cmd.sh
#!/bin/sh

echo "You are '$(whoami)'."  
echo "I'm going to store that in a file for you at '${APP_WORKDIR}/whoami.txt'."

echo $(whoami) > "${APP_WORKDIR}/whoami.txt"

echo "Here's the permissions:"  
ls -lah "${APP_WORKDIR}/whoami.txt"  

Running

Now that we have all of the pieces of this put together, we can build it, and see how it actually runs. I'm going to be testing this on a Ubuntu Xenial VPS. I've set up a user called 'services' with the UID and GID of 1000. I'm using a Linux environment because that way I'm able to directly test the permissions that files get written as when written in mounted volumes without something else getting in the way (like shared folders or NFS that can hide issues with permissions).

First off, we have to build the image:

services@tasha:~/pdvp$ docker build .  
Sending build context to Docker daemon 7.168 kB  
Step 1 : FROM alpine:3.4  
 ---> 4e38e38c8ce0
...
Successfully built e9331fa9bebe  

Then we will run it. We'll mount the current directory in the predefined working directory of the image so that the file that our CMD script writes is written into our current directory. This way we can see the permissions that the file has both in and out of the container. We are aiming to have the file be created by app:app in the container, and be owned by services:services outside the container.

services@tasha:~/pdvp$ docker run --rm -v `pwd`:/opt/app e9331fa9bebe  
==> Attempting to create 'app' user.
    Creating 'app' user with UID:GID of 1000:1000
==> Attempting to create home directory at '/home/app'.
    Done!
==> Attempting to create working directory at '/opt/app'.
    Already exists!
==> Running CMD as 'app'...
You are 'app'.  
I'm going to store that in a file for you at '/opt/app/whoami.txt'.  
Here's the permissions:  
-rw-r--r--    1 app      app            4 Sep 14 20:20 /opt/app/whoami.txt

Alright! We have created a file as app:app within the container. We can see that even though we started this container as root, the user we are actually running the command as is indeed app, as we wanted. The file has been created successfully thanks to the folder that is volume mounted from the host being owned by a user with the same UID and GID. Now one final test, checking the permissions on the host:

services@tasha:~/pdvp$ ll whoami.txt  
-rw-r--r-- 1 services services 4 Sep 14 20:02 whoami.txt

As expected, the file is owned by services:services.

One final thing... what if we specify a UID:GID combination that doesn't exist on the host? That goes like this:

services@tasha:~/pdvp$ docker run --rm -e APP_UID=1234 -e APP_GID=1234 -v `pwd`:/opt/app e9331fa9bebe  
==> Attempting to create 'app' user.
    Creating 'app' user with UID:GID of 1234:1234
==> Attempting to create home directory at '/home/app'.
    Done!
==> Attempting to create working directory at '/opt/app'.
    Already exists!
==> Running CMD as 'app'...
You are 'app'.  
I'm going to store that in a file for you at '/opt/app/whoami.txt'.  
/docker-cmd.sh: line 6: can't create /opt/app/whoami.txt: Permission denied
Here's the permissions:  
ls: /opt/app/whoami.txt: No such file or directory  

As you can see... it doesn't work. However, if you create a directory with the UID and GID set to whatever you want to run the container as, it will work:

services@tasha:~/pdvp$ ll ./  
...
drwxrwxr-x 2     1234     1234 4096 Sep 14 20:22 u1234g1234/  
services@tasha:~/pdvp$ docker run --rm -e APP_UID=1234 -e APP_GID=1234 -v `pwd`/u1234g1234:/opt/app e9331fa9bebe  
==> Attempting to create 'app' user.
    Creating 'app' user with UID:GID of 1234:1234
==> Attempting to create home directory at '/home/app'.
    Done!
==> Attempting to create working directory at '/opt/app'.
    Already exists!
==> Running CMD as 'app'...
You are 'app'.  
I'm going to store that in a file for you at '/opt/app/whoami.txt'.  
Here's the permissions:  
-rw-r--r--    1 app      app            4 Sep 14 20:23 /opt/app/whoami.txt
services@tasha:~/pdvp$ ll u1234g1234/whoami.txt  
-rw-r--r-- 1     1234     1234    4 Sep 14 20:23 whoami.txt

Caveats

The only issue I've found so far with this approach is that since you have to run the container with it's USER instruction left as the default root, when you aren't using $ docker run (e.g. when you're using $ docker exec) you will not be going through the ENTRYPOINT, and thus not end up as the right user. In most cases though you can easily get around this. For example, $ docker exec has the -u flag that you can use to specify a user to execute a command as. For example:

$ docker exec -u 1000:1000 test whoami
whoami: cannot find name for user ID 1000  

NB: The format for -u|--user is <name|uid>[:<group|gid>]. You can specify a name, but if it doesn't exist it will fail.

Coming Full Circle

As you can see we've accomplished everything we set up to achieve. We have a very configurable, lightweight container that is very flexible in it's usage. This approach is in use on many developer machines, and production at Byng.

Really, I'd love to see Docker have some native, built-in, cross-platform solution to this problem. Perhaps Docker for Windows/Mac will help move towards that solution by unifying the different platforms with a set of official Docker tools for all platforms. At present though they don't quite live up to the alternatives, and that opens up the potential for plenty of differences in people's environments. In any case, hopefully this post has helped you hear the words "it works on my machine" at least a little less often.