docker-compose-config-skill/SKILL.md

---
name: docker-compose-config
description: Docker Compose configuration management for multi-host server deployments. Use when creating, modifying, or managing docker-compose.yml files with environment-specific configurations, external volume mounts, and reverse proxy networks. Triggers on tasks involving Docker Compose files, environment overrides, multi-host deployments, or service configuration for self-hosted applications.
---

# Docker Compose Configuration Management

This skill provides guidance for managing Docker Compose configurations across multiple server environments with per-host overrides.

## Directory Structure

Each project follows this structure:

```
/docker/config/
├── <project>/
│   ├── docker-compose.yml              # Main service definitions
│   ├── docker-compose.override.yml     # Current host overrides (gitignored)
│   ├── .env                            # Environment variables (gitignored)
│   ├── .gitignore                      # Excludes /.env and /docker-compose.override.yml
│   ├── env.example                     # Template for .env files
│   ├── README.md                       # Setup and usage instructions
│   └── environments/
│       └── <hostname>/                 # Per-host configs (committed, secrets scrubbed)
│           ├── .env
│           └── docker-compose.override.<hostname>.yml
```

## Compose File Conventions

### Service Definition Pattern

```yaml
name: <project-name>

services:
  <service-name>:
    container_name: <service_container>
    image: <registry>/<image>:<version>
    environment:
      PUID: ${UID}
      PGID: ${GID}
      # Service-specific vars use env substitution
      VAR_NAME: ${VAR_NAME}
    volumes:
      - ${DATA_PATH}:/app/data
      - /etc/localtime:/etc/localtime:ro
    ports:
      - ${SERVICE_PORT:-default}:internal_port
    restart: unless-stopped
    networks:
      - default
      - reverse_proxy

networks:
  reverse_proxy:
    name: reverse_proxy
    attachable: true
```

### Key Patterns

1. **Image versioning**: Always use pinned version tags (never `latest`)
   ```yaml
   image: ghcr.io/org/image:v1.0.0
   image: lscr.io/linuxserver/app:2.5.1
   ```
   Note: SHA256 digests are managed automatically by Renovate bot and should not be added manually.

2. **Port defaults**: Always provide defaults for ports
   ```yaml
   ports:
     - ${APP_PORT:-8080}:8080
   ```

3. **Volume mounts**: Use environment variables for paths
   ```yaml
   volumes:
     - ${UPLOAD_LOCATION}:/usr/src/app/upload
     - ${DB_DATA_LOCATION}/data:/var/lib/postgresql/data
   ```

4. **Proxy network**: Services needing reverse proxy access join `reverse_proxy` network
   ```yaml
   networks:
     - default
     - reverse_proxy

   networks:
     reverse_proxy:
       name: reverse_proxy
       attachable: true
   ```

## Environment Variable Isolation

**Never use `env_file` directives in service definitions.** Each service should only receive the specific environment variables it needs.

### Why
- **Security**: Prevents secrets meant for one service from leaking to others
- **Clarity**: Makes explicit which variables each service requires
- **Debugging**: Easier to trace environment-related issues

### Correct Pattern
```yaml
services:
  app:
    environment:
      - DATABASE_URL=postgres://${DB_USER}:${DB_PASS}@db:5432/${DB_NAME}
      - APP_SECRET=${APP_SECRET}
    # NO env_file directive

  database:
    environment:
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASS}
      POSTGRES_DB: ${DB_NAME}
    # NO env_file directive
```

### Incorrect Pattern
```yaml
services:
  app:
    env_file:
      - .env  # BAD: passes ALL variables to container
    environment:
      - EXTRA_VAR=value

  database:
    env_file:
      - .env  # BAD: database receives app secrets, TURN secrets, etc.
```

### How .env Files Work
The `.env` file serves **compose-time interpolation only**:
- Variables like `${DATA_PATH}` in volumes are substituted when `docker compose` parses the file
- This happens at compose parse time, NOT at container runtime
- The container only receives variables explicitly listed in `environment:`

## Environment File Conventions

### Structure (.env)

```bash
# Permissions
UID=1000
GID=1000

# Paths
UPLOAD_LOCATION=/mnt/user/pictures
DB_DATA_LOCATION=/mnt/user/appdata/<project>

# Ports
APP_PORT=8080

# Secrets
DB_PASSWORD=<generated>

# Database
DB_HOSTNAME=database
DB_USERNAME=postgres
DB_DATABASE_NAME=<project>
```

### Per-Host Variations

- `environments/<hostname>/.env` overrides paths for specific hosts
- Common overrides: mount paths (`/mnt/user/` vs `/mnt/main/`), ports

## Secret Management with SOPS

Projects use SOPS with age encryption to securely commit secrets to git. This replaces the manual "secret scrubbing" approach.

### Directory Structure (with SOPS)

```
/docker/config/<project>/
├── docker-compose.yml
├── docker-compose.override.yml     # (gitignored)
├── .env                            # (gitignored, generated by decrypt)
├── .gitignore
├── .sops.yaml                      # SOPS encryption config
├── Makefile                        # encrypt/decrypt targets
├── env.sops.yaml                   # Encrypted env vars (committed)
└── environments/
    └── <hostname>/
        ├── env.sops.yaml           # Encrypted, host-specific (committed)
        └── .env                    # (gitignored, generated)
```

### Key Repository

Public keys are managed centrally at `~/repos/sops-age-keys/`:
- `recipients/*.pub` - Public keys for each host
- `scripts/init-host.sh` - Generate key for new host
- `scripts/add-recipient.sh` - Add host's public key
- `templates/Makefile` - Makefile template for projects

Private keys stay at `~/.config/sops/age/keys.txt` (never committed).

### .sops.yaml Configuration

```yaml
creation_rules:
  - path_regex: ".*"
    encrypted_regex: "^(.*PASSWORD.*|.*SECRET.*|.*KEY.*|.*PASS.*)$"
    age: age1abc...
```

- `path_regex: ".*"` matches any file to avoid "no matching creation rules" errors
- `encrypted_regex` specifies which keys to encrypt (only secrets)
- Get recipient keys from `~/repos/sops-age-keys/recipients/*.pub`
- Comments and non-matching keys remain unencrypted

### env.sops.yaml Format

Use normal variable names. Only keys matching `encrypted_regex` are encrypted:

```yaml
# Permissions
UID: "1000"
GID: "1000"

# Paths
UPLOAD_LOCATION: /mnt/user/pictures
DB_DATA_LOCATION: /mnt/user/appdata/immich

# Ports
APP_PORT: "2283"

# Database
DB_HOSTNAME: database
DB_USERNAME: postgres
DB_DATABASE_NAME: immich
DB_PASSWORD: secret_value_here
```

After encryption, only `DB_PASSWORD` is encrypted; comments and other values remain readable.

### Makefile Targets

```makefile
decrypt:
	@sops decrypt --input-type dotenv --output-type dotenv --output .env env.sops.yaml
	@echo "Decrypted env.sops.yaml -> .env"

encrypt:
	@sops encrypt --input-type dotenv --output env.sops.yaml .env
	@echo "Encrypted .env -> env.sops.yaml"

edit:
	@sops --input-type dotenv --output-type dotenv env.sops.yaml

clean:
	@rm -f .env
	@echo "Removed .env"
```

### Common SOPS Commands

```bash
# Copy encrypted file from environment to project root
cp environments/<host>/env.sops.yaml env.sops.yaml

# Decrypt to .env for editing
make decrypt

# Edit the .env file
nano .env

# Re-encrypt after changes
make encrypt

# Copy back to environment
cp env.sops.yaml environments/<host>/env.sops.yaml

# Clean up
make clean

# Edit secrets directly (decrypts in editor, re-encrypts on save)
make edit

# Add new host to recipients (after adding to .sops.yaml)
sops updatekeys env.sops.yaml
```

### Starting Services with SOPS

```bash
cd /docker/config/<project>

# Copy and decrypt secrets to project root
cp environments/<host>/env.sops.yaml env.sops.yaml
make decrypt

# Start services
docker compose -f docker-compose.yml \
  -f environments/<host>/docker-compose.override.<host>.yml up -d

# Clean up (optional)
make clean && rm env.sops.yaml
```

### Migrating Existing Projects

See `~/repos/sops-age-keys/README.md` for full migration guide, or `~/repos/sops-age-keys/docs/design.md` for architecture details.

## Common Commands

```bash
# Start a project
docker compose -f <project>/docker-compose.yml up -d

# Start with host-specific override
docker compose -f <project>/docker-compose.yml \
  -f <project>/environments/<host>/docker-compose.override.<host>.yml up -d

# View logs
docker compose -f <project>/docker-compose.yml logs -f

# Stop services
docker compose -f <project>/docker-compose.yml down
```

## Deploying Updates from Merged PRs

After merging PRs that update image versions (e.g., Renovate bot PRs), deploy the changes to the target host. To review and merge Renovate PRs, use the gitea skill's PR management functions (`gitea_list_renovate_prs`, `gitea_merge_pr`).

### Pull and Redeploy a Stack

```bash
# On the target host, pull latest config and redeploy
ssh bill@<host> "cd /docker/config/<stack-name> && git pull origin main"

# Pull new images and restart
ssh bill@<host> "cd /docker/config/<stack-name> && \
  docker compose pull && \
  docker compose up -d"

# For stacks with host-specific overrides:
ssh bill@<host> "cd /docker/config/<stack-name> && \
  docker compose -f docker-compose.yml \
  -f environments/<host>/docker-compose.override.<host>.yml \
  pull && \
  docker compose -f docker-compose.yml \
  -f environments/<host>/docker-compose.override.<host>.yml \
  up -d"
```

### Verify Deployment

```bash
# Check container is running with new image
ssh bill@<host> "docker ps --filter name=<container> --format '{{.Image}} {{.Status}}'"

# Check logs for startup errors
ssh bill@<host> "docker logs --tail 20 <container>"
```

### Host Reference

| Host | SSH | Stack Path |
|------|-----|------------|
| inkling | `bill@inkling` | `/docker/config/<stack>/` |
| shellington | `bill@shellington` | `/docker/config/<stack>/` |

## Creating a New Project

When creating a new Docker Compose project:

1. Create the project directory structure
2. Create `docker-compose.yml` with service definitions
3. Create `env.example` as a template
4. Create environment config **only for the current host** (`environments/<current-hostname>/.env`)
5. Optionally create a `README.md` with first-run instructions

**Important**: Only create the environment for the current host unless explicitly asked to create configurations for other hosts.

## Adding a New Host Environment

1. Create `environments/<hostname>/` directory
2. Copy existing host's `.env` as template
3. Update paths and port mappings for the host
4. Create override compose file if device mappings differ

## Git Repository Management

Each project should be version controlled with its own git repository.

### .gitignore Configuration

```gitignore
# Root environment file (generated, may contain secrets)
/.env

# Docker compose override (host-specific, not committed)
/docker-compose.override.yml

# Per-host decrypted .env files (generated from env.sops.yaml)
/environments/*/.env
```

**Note**: All `.env` files are gitignored. Secrets are stored encrypted in `env.sops.yaml` files using SOPS (see "Secret Management with SOPS" section above).

### Creating a Git Repository

```bash
# Initialize
cd /docker/config/<project>
git init && git branch -m main

# Create remote on Gitea (using git-gitea skill)
source ~/.claude/skills/git-gitea/scripts/gitea-helper.sh
gitea_create_repo "docker-<project>" "Docker Compose configuration for <project>" true

# Add remote, commit, push
git remote add origin https://git.prettyhefty.com/Bill/docker-<project>.git
git add -A
git commit -m "Initial commit: <project> docker-compose configuration"
git push -u origin main
```

## Service Dependencies

Use `depends_on` with health checks for proper startup order:

```yaml
depends_on:
  database:
    condition: service_healthy
```

## Healthchecks

```yaml
healthcheck:
  test: ["CMD", "curl", "-f", "http://127.0.0.1:8080/"]
  start_period: 5m
  interval: 30s
  timeout: 20s
  retries: 10
```

## References

- See [examples.md](references/examples.md) for complete service configurations
- See [networks.md](references/networks.md) for network topology details