Posted: 21/06/2023
Docker Compose Parallel Gotcha
Time for another Gotcha with Docker.
This time, we kept seeing calls in Azure DevOps to
docker compose up
that would never complete. It would start some of the services (We have about 40), and then it would just hang.
We have recently upgraded to Docker Compose V2 due to V1 being retired.
We did not see this issue on our local machines, only on DevOps.
The only thing that was different (apart from
docker-compose
changing to
docker compose
in our scripts) is that we noticed during a local build (docker compose build) of our containers, that we would swamp our PCs and make them unusable during the build process, this meant we added this directive to our environments:
COMPOSE_PARALLEL_LIMIT=2
This solved the issue, and we thought nothing of it. Several weeks later, this issue occurs.
Lots of researching of this docker compose hanging, lead to this issue about lacking "entropy" on a build agent:
https://stackoverflow.com/questions/59941911/docker-compose-up-hangs-forever-how-to-debug
https://akliang.medium.com/docker-compose-and-entropy-5b3f00561b88
None of the suggestions there helped.
Then I noticed that if we run a
docker compose up
with a specific container, it never hangs.
This led me to this GitHub issue:
https://github.com/docker/compose/issues/7486
And again the
COMPOSE_PARALLEL_LIMIT
being a factor.
DAMMIT!!
So we see that having it too low, stops
compose
from being able to start (note that this is not consistent, sometimes it works fine 5 or 6 times in a row)
So we set it to 200 in our env file:
COMPOSE_PARALLEL_LIMIT=200
and now it all works fine!
I will add an additional .env file with an override for a local build to help the developers, but for now devops is working.
Thanks for reading this post.
If you want to reach out, catch me on Twitter!
I am always open to mentoring people, so get in touch.