Run a background script inside a docker container

Question

In my Dockerfile, I run a script:

RUN /bin/sh -c scripts/init.sh

Inside init.sh, all commands ending with & are not executed: I cannot run background processes. Any idea why?

Answer 1

I had the similar issue and something like the following helped me.

RUN nohup bash -c "scripts/init.sh &" && sleep 4

In many cases the server you started isn’t yet fully ready. To allow the server a bit more time to get ready add a sleep command. How large the argument sleep needs to depend on the service you start and you probably need to tweak it.

Read more on this Doc

Answer 2

The reason why it doesn't seem to work to run commands in the background with docker RUN commands is that RUN happens at docker build time. These commands help build the docker image, such as installing the things that need to be in the docker when it is instantiated, or setting up configuration files that will be used when the docker is instantiated. It makes no sense to run commands in the background at build time.

Answer 3

To run something in the background, let's use this in a Dockerfile:

FROM ubuntu:20.04
RUN mkdir /scripts
RUN echo 'touch /tmp/file &' >> /scripts/init.sh
RUN echo 'tail -F /dev/null' >> /scripts/init.sh
CMD sh -c "sh /scripts/init.sh"

You can replace tail -F /dev/null with the next command/service which doesn't exit and keeps container running.

Building and testing:

docker build -t mytest .
docker run -td --name mytest mytest
docker exec mytest ls -l /tmp/file

A note that entrypoint and command should not be set in Docker Compose, Kubernetes or any container management/orchestration tool. They override Dockerfile's CMD. Or you can set same CMD. So CMD is used at execution time, while RUN is used only when building image.

---

To make best quality app, you have to put your task to another container. For instance, put it to init container, use CronJob, Job or something equivalent.

Of course, tasks are still fine to be executed in background in any programming language.

But specifically Shell needs special efforts.

---

So here is bonus part

I have tested this only in Ubuntu, a thread logging using Shell and Docker.

Try adding this step after another touch step:

RUN echo 'touch /proc/no_file_can_be_created_in_proc_dir > /proc/1/fd/1 >> /proc/1/fd/2 &' >> /scripts/init.sh

So it prints logs for that background task:

$ docker logs mytest
touch: cannot touch '/proc/no_file_can_be_created_in_proc_dir': No such file or directory

So the trick is to use > /proc/1/fd/1 >> /proc/1/fd/2 & instead of just & in your command.

Answer 4

Your scripts do run in the background, however only for the duration of that particular RUN command. Running processes are not part of the state of an image layer, and each RUN command builds a new image layer. It's not a virtual machine running each command in sequence and keeping all processes running in between. It's better to think of each layer as a folder of files, containing the consequences of all commands up to that point.

As others have pointed out, running multiple background processes is outside the spirit of what a single container is intended to do; it is better to run them as separate containers and orchestrate them with docker-compose or kubernetes.

However, you can do it by running both your background processes and your "foreground process" (the main purpose of your image) as part of the CMD statement at the end of your Dockerfile:

# Start MongoDB in the background, run the application in the foreground
CMD bash -c "mongod --dbpath=/root/tmp-mongodb-data & sleep 5 && npm run start"

The sleep command, as others have suggested, is there to wait for the background process (in this case MongoDB) to be ready to talk to the foreground process. Of course there are better ways, like a script that waits for a port to actually start taking connections.

However, in my case I didn't need MongoDB for the final foreground command — I needed it for one of my intermediate RUN commands that creates a layer. And, you can solve that too:

RUN bash -c "mongod --dbpath=/root/tmp-mongodb-data & sleep 5 && npm run build"

The key here is to start the background process separately for each RUN or CMD statement that must have access to it.

(To be clear, generally there is only one CMD statement at the end, they are not interchangeable with RUN statements.)

Answer 5

My first idea is to create services inside the container instead of running them with nohup or &, run them as system service and you don't need to handle them in init.sh.

But this is not a "real" docker approach. If you need more than 1 service to run, separate them to different containers (1 container - 1 service) and put all of them together with a docker-compose solution.