How to fix "Permission Denied" when manipulating files in Docker container

You're chugging along with your CI/CD pipeline (perhaps using GitHub Actions), deploying your application with Docker. Suddenly, your Python service, which needs to create directories using os.makedirs within a mounted volume, throws a dreaded "Permission Denied" error. This is a common headache when Docker containers interact with the host filesystem, particularly for writing operations.

The culprit? A mismatch between the user running the process inside your Docker container and the user who owns the files on the host machine (e.g., the actions-runner user in GitHub Actions).

This is especially common when running os.makedirs or os.chmod, ...

Here's a straightforward guide to resolve this.

The Problem: User Mismatch with Bind Mounts

When you use a Docker bind mount (e.g., .:/workspace in your compose.yml), you're essentially telling Docker: "Make this directory from my host machine available inside the container at this path."

The key issue is that files and directories within this bind mount retain their original ownership and permissions from the host system.

1. Host File Ownership: In a CI environment like GitHub Actions, the checked-out code (your workspace) is typically owned by a specific user (e.g., actions-runner with UID 1001, GID 1001).
2. Container User: Your Docker container, by default or due to a USER instruction in its Dockerfile, might run its main process as a different user (e.g., root - UID 0, or another user like UID 1000).
3. The Clash: When the process inside the container (e.g., running as UID 0 or 1000) tries to create a directory (os.makedirs) within the mounted /workspace (which is owned by UID 1001 on the host), the operating system on the host enforces its permissions. If user 1000 doesn't have write permission to a directory owned by user 1001, you get "Permission Denied."

The Quick Fix: Aligning Container User with Host User

The most effective solution is to tell Docker to run the process inside your container with the same User ID (UID) and Group ID (GID) as the user who owns the files on the host.

You can achieve this by adding the user directive to your service definition in your compose.yml file.

How to Implement the Fix

Step 1: Modify Your compose.yml

For each service that needs to write to a bind-mounted volume, add the user directive:

version: '3.8' # Or your preferred version

services:
  your_service_name: # e.g., backend, worker, app
    image: your_image_name
    # ... other configurations ...
    volumes:
      - ./your_local_code_or_data:/app/data # Example bind mount
    user: "${UID:-0}:${GID:-0}" # The magic line!
    # ... other configurations ...

  # Potentially other services
  # worker:
  #   image: your_worker_image
  #   volumes:
  #     - ./worker_data:/app/worker_files
  #   user: "${UID:-0}:${GID:-0}"

What user: "${UID:-0}:${GID:-0}" Does:

• user:: This Docker Compose directive specifies the UID and GID to run the container's command as.
• $UID and $GID: These are environment variables that Docker Compose expects to be present in the shell environment where you run compose up. You need to set them to the UID and GID of the host user.
• ${ ... :-0}: This is shell parameter expansion.
  • If UID (or GID) is set and not empty, its value is used.
  • Otherwise (if UID or GID is unset or empty), it defaults to 0. UID 0 and GID 0 represent the root user.

Why This Works

By setting user: "${UID}:${GID}" and ensuring UID and GID match the host user who owns the workspace (e.g., actions-runner), you achieve harmony:

1. Aligned Ownership: The process inside the container now runs with the same UID and GID as the owner of the files on the host system.
2. Correct Permissions: From the operating system's perspective, the process (e.g., UID 1001) inside the container is the owner of the files (owned by UID 1001) in the mounted volume.
3. No More Denials: Standard owner permissions (e.g., read, write, execute for the owner) apply correctly, and os.makedirs can create directories as intended.
4. Correct File Creation Ownership: Any new files or directories created by the container within the mounted volume will now be owned by the host user (e.g., actions-runner), which is usually the desired behavior in CI pipelines and for local development.

What if UID/GID are NOT set (The :-0 Fallback)?

If, for some reason, the UID and GID environment variables are not set or are empty when compose up is executed, the ${UID:-0}:${GID:-0} will default to 0:0. This means your container process will run as root.

• Pros (for this specific error): Running as root inside the container will generally allow it to write anywhere, including to host directories owned by root:root or other users. This might "fix" the permission error if your host directories were root:root and the container previously ran as non-root.
• Cons: Files created by the container will be owned by root on the host. This can be problematic for subsequent CI steps or for managing these files on the host later. The goal is generally to match the host user, not necessarily to become root.

When is This Technique Most Relevant?

This user: "${UID:-0}:${GID:-0}" approach is particularly vital when:

• Using CI/CD systems (like GitHub Actions, GitLab CI, Jenkins) where Docker containers need to write to the checked-out workspace, which is owned by a specific runner user.
• Local development where you want files generated by containers (e.g., build artifacts, logs, uploaded files) to be owned by your local user, not root.
• Any scenario where Docker containers perform filesystem manipulations (create, modify files/directories) within bind-mounted volumes.

By implementing this fix, you can ensure smoother Docker operations, prevent frustrating permission errors, and maintain correct file ownership across your development and deployment environments.