Initial commit: docker-service-architecture skill
Comprehensive skill for Docker containerization patterns including: - Multi-service project organization (services/<name>/) - Environment-specific deployments (dev/test/prod) - 3-stage testing pipeline (unit/service/integration) - Branch-isolated test environments with dynamic ports - CI/CD pipeline with semantic versioning - Rich test runner with context-efficient output
This commit is contained in:
405
SKILL.md
Normal file
405
SKILL.md
Normal file
@@ -0,0 +1,405 @@
|
||||
---
|
||||
name: docker-service-architecture
|
||||
description: Use when setting up a new project with multiple services that need Docker containerization, multi-environment deployments (dev/test/prod), CI/CD pipelines, or multi-stage testing (unit/service/integration). Also use when adding testing infrastructure to existing Docker projects.
|
||||
---
|
||||
|
||||
# Docker Service Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
Pattern for organizing multi-service projects with Docker containerization, environment-specific deployments, and a 3-stage testing pipeline. Key principles: **service isolation**, **ephemeral test environments**, **branch-isolated testing**, and **context-efficient test output**.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Setting up a new multi-service project
|
||||
- Adding Docker containerization to existing services
|
||||
- Creating CI/CD pipelines for containerized services
|
||||
- Implementing multi-stage testing (unit → service → integration)
|
||||
- Need parallel test execution across git branches
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
project/
|
||||
├── services/
|
||||
│ ├── service-a/
|
||||
│ │ ├── Dockerfile
|
||||
│ │ ├── pyproject.toml (or package.json)
|
||||
│ │ ├── service_a/ # Source code
|
||||
│ │ ├── tests/
|
||||
│ │ │ ├── unit/ # No containers needed
|
||||
│ │ │ └── integration/ # Needs service containers
|
||||
│ │ └── deploy/
|
||||
│ │ └── test/
|
||||
│ │ └── docker-compose.yml # Service-isolated tests
|
||||
│ └── service-b/
|
||||
│ └── ... (same structure)
|
||||
├── deploy/
|
||||
│ ├── dev/
|
||||
│ │ ├── docker-compose.yml
|
||||
│ │ └── .env.example
|
||||
│ ├── test/
|
||||
│ │ └── docker-compose.yml # Full-stack integration
|
||||
│ └── prod/
|
||||
│ ├── docker-compose.yml
|
||||
│ └── .env.example
|
||||
├── scripts/
|
||||
│ ├── test-runner.py # Rich progress display
|
||||
│ ├── test-service.sh # Service-level test runner
|
||||
│ ├── run-integration-tests.sh
|
||||
│ └── get-test-instance-id.sh
|
||||
├── .gitea/workflows/ # Or .github/workflows/
|
||||
│ └── release.yml
|
||||
└── Makefile
|
||||
```
|
||||
|
||||
## 3-Stage Testing Pattern
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────────┐
|
||||
│ Unit Tests │ ──► │Service Tests│ ──► │Integration Tests│
|
||||
│ (no Docker) │ │(per-service)│ │ (full stack) │
|
||||
└─────────────┘ └─────────────┘ └─────────────────┘
|
||||
Fast Medium Slow
|
||||
Mocked deps Real DB, mocked All services
|
||||
external APIs together
|
||||
```
|
||||
|
||||
**Fail-fast**: Each stage only runs if previous stage passes.
|
||||
|
||||
### Stage 1: Unit Tests
|
||||
|
||||
- **Location**: `services/<name>/tests/unit/`
|
||||
- **Dependencies**: All mocked
|
||||
- **Containers**: None
|
||||
- **Speed**: Seconds
|
||||
|
||||
### Stage 2: Service Tests
|
||||
|
||||
- **Location**: `services/<name>/tests/integration/`
|
||||
- **Config**: `services/<name>/deploy/test/docker-compose.yml`
|
||||
- **Dependencies**: Real database, mocked external APIs
|
||||
- **Containers**: Service + its direct dependencies only
|
||||
- **Speed**: 30-60 seconds per service
|
||||
|
||||
### Stage 3: Integration Tests
|
||||
|
||||
- **Location**: `tests/integration/` or `services/<name>/tests/integration/`
|
||||
- **Config**: `deploy/test/docker-compose.yml`
|
||||
- **Dependencies**: All services running
|
||||
- **Containers**: Full stack
|
||||
- **Speed**: 1-3 minutes
|
||||
|
||||
## Test Environment Isolation
|
||||
|
||||
### Branch Isolation
|
||||
|
||||
Use `TEST_INSTANCE_ID` derived from git branch for parallel testing:
|
||||
|
||||
```bash
|
||||
# scripts/get-test-instance-id.sh
|
||||
#!/bin/bash
|
||||
BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown")
|
||||
# Sanitize: replace non-alphanumeric with dash, limit length
|
||||
echo "$BRANCH" | sed 's/[^a-zA-Z0-9]/-/g' | cut -c1-20
|
||||
```
|
||||
|
||||
### Docker Compose with Instance ID
|
||||
|
||||
```yaml
|
||||
# deploy/test/docker-compose.yml
|
||||
services:
|
||||
database:
|
||||
container_name: mydb-test-${TEST_INSTANCE_ID}
|
||||
# Use tmpfs for ephemeral storage (fast, stateless)
|
||||
tmpfs:
|
||||
- /var/lib/postgresql/data
|
||||
# Dynamic ports - no conflicts
|
||||
ports:
|
||||
- "5432" # Docker assigns random host port
|
||||
networks:
|
||||
- test-network
|
||||
|
||||
networks:
|
||||
test-network:
|
||||
name: myproject-test-${TEST_INSTANCE_ID}
|
||||
```
|
||||
|
||||
### Dynamic Port Discovery
|
||||
|
||||
```bash
|
||||
# After containers start, discover assigned ports
|
||||
API_PORT=$(docker compose port api 8000 | cut -d: -f2)
|
||||
DB_PORT=$(docker compose port database 5432 | cut -d: -f2)
|
||||
|
||||
export TEST_API_URL="http://localhost:${API_PORT}"
|
||||
export TEST_DATABASE_URL="postgresql://test:test@localhost:${DB_PORT}/testdb"
|
||||
```
|
||||
|
||||
## Docker Compose Patterns
|
||||
|
||||
### Development Environment
|
||||
|
||||
```yaml
|
||||
# deploy/dev/docker-compose.yml
|
||||
services:
|
||||
api:
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: services/api/Dockerfile
|
||||
ports:
|
||||
- "${API_PORT:-8000}:8000" # Configurable external port
|
||||
volumes:
|
||||
# Mount source for hot-reload
|
||||
- ../../services/api/src:/app/src:ro
|
||||
depends_on:
|
||||
database:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
|
||||
volumes:
|
||||
postgres-data:
|
||||
name: myproject-dev-postgres # Named volumes persist
|
||||
```
|
||||
|
||||
### Test Environment
|
||||
|
||||
```yaml
|
||||
# deploy/test/docker-compose.yml
|
||||
services:
|
||||
database:
|
||||
# Ephemeral storage for speed
|
||||
tmpfs:
|
||||
- /var/lib/postgresql/data
|
||||
# PostgreSQL tuning for tests
|
||||
command: >
|
||||
postgres
|
||||
-c synchronous_commit=off
|
||||
-c fsync=off
|
||||
-c full_page_writes=off
|
||||
healthcheck:
|
||||
interval: 5s # Faster checks for tests
|
||||
retries: 10
|
||||
start_period: 10s
|
||||
```
|
||||
|
||||
### Production Environment
|
||||
|
||||
```yaml
|
||||
# deploy/prod/docker-compose.yml
|
||||
services:
|
||||
api:
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 4G
|
||||
cpus: "4"
|
||||
reservations:
|
||||
memory: 1G
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-size: "100m"
|
||||
max-file: "10"
|
||||
```
|
||||
|
||||
## CI/CD Pipeline
|
||||
|
||||
### Release Workflow (Gitea/GitHub Actions)
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/release.yml
|
||||
name: Release
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- 'v[0-9]+.[0-9]+.[0-9]+*'
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:15
|
||||
env:
|
||||
POSTGRES_PASSWORD: test
|
||||
options: --health-cmd pg_isready
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Run tests
|
||||
run: make test
|
||||
|
||||
build:
|
||||
needs: test # Only build if tests pass
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
service: [api, worker, frontend] # Build all services
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Extract version
|
||||
id: version
|
||||
run: |
|
||||
VERSION=${GITHUB_REF#refs/tags/v}
|
||||
echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
|
||||
# Detect pre-release (contains hyphen: v1.0.0-beta)
|
||||
[[ "$VERSION" == *-* ]] && echo "PRERELEASE=true" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Build and push
|
||||
run: |
|
||||
IMAGE=registry.example.com/myproject-${{ matrix.service }}
|
||||
docker build -t $IMAGE:${{ steps.version.outputs.VERSION }} \
|
||||
-f services/${{ matrix.service }}/Dockerfile .
|
||||
docker push $IMAGE:${{ steps.version.outputs.VERSION }}
|
||||
|
||||
# Only tag 'latest' for stable releases
|
||||
if [ "${{ steps.version.outputs.PRERELEASE }}" != "true" ]; then
|
||||
docker tag $IMAGE:${{ steps.version.outputs.VERSION }} $IMAGE:latest
|
||||
docker push $IMAGE:latest
|
||||
fi
|
||||
```
|
||||
|
||||
## Rich Test Runner
|
||||
|
||||
Context-efficient test output with real-time progress:
|
||||
|
||||
```python
|
||||
# scripts/test-runner.py (simplified structure)
|
||||
from rich.console import Console
|
||||
from rich.live import Live
|
||||
from rich.table import Table
|
||||
|
||||
class TestRunner:
|
||||
def run_all(self, unit=True, service=True, integration=True):
|
||||
with Live(console=self.console) as live:
|
||||
if unit:
|
||||
self.run_stage("Unit Tests", self.unit_targets)
|
||||
if not self.all_passed:
|
||||
return False # Fail fast
|
||||
|
||||
if service:
|
||||
self.run_stage("Service Tests", self.service_targets)
|
||||
if not self.all_passed:
|
||||
return False
|
||||
|
||||
if integration:
|
||||
self.run_stage("Integration Tests", self.integration_targets)
|
||||
|
||||
self.render_summary()
|
||||
return self.all_passed
|
||||
```
|
||||
|
||||
### Output Parsing
|
||||
|
||||
```python
|
||||
# Parse pytest progress: "tests/test_foo.py::test_bar PASSED [ 45%]"
|
||||
PYTEST_PROGRESS = re.compile(r"\[\s*(\d+)%\]")
|
||||
PYTEST_SUMMARY = re.compile(r"(\d+) passed(?:.*?(\d+) failed)?")
|
||||
|
||||
# Parse vitest: "✓ src/foo.test.tsx (11 tests) 297ms"
|
||||
VITEST_PASS = re.compile(r"✓\s+(.+\.tsx?)\s+\((\d+)\s+test")
|
||||
```
|
||||
|
||||
### Display Format
|
||||
|
||||
```
|
||||
Unit Tests
|
||||
● price-db ━━━━━━━━━━░░░░░░░░░░ 52% 497/955 12.3s
|
||||
→ test_data_quality_validation
|
||||
✓ orchestrator 74/74 3.2s
|
||||
○ frontend pending
|
||||
|
||||
Service Tests
|
||||
○ price-db pending
|
||||
○ orchestrator pending
|
||||
```
|
||||
|
||||
## Makefile Integration
|
||||
|
||||
```makefile
|
||||
VERBOSE ?= 0
|
||||
|
||||
test:
|
||||
@python scripts/test-runner.py $(if $(filter 1,$(VERBOSE)),-v)
|
||||
|
||||
test-verbose:
|
||||
@python scripts/test-runner.py -v
|
||||
|
||||
test-unit:
|
||||
@$(MAKE) -C services/api test-unit VERBOSE=$(VERBOSE)
|
||||
@$(MAKE) -C services/worker test-unit VERBOSE=$(VERBOSE)
|
||||
|
||||
test-service SERVICE:
|
||||
@./scripts/test-service.sh $(SERVICE)
|
||||
|
||||
test-integration:
|
||||
@./scripts/run-integration-tests.sh
|
||||
|
||||
test-infra-up:
|
||||
@cd deploy/test && docker compose up -d --build --wait
|
||||
|
||||
test-infra-down:
|
||||
@cd deploy/test && docker compose down -v
|
||||
```
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
| Mistake | Problem | Solution |
|
||||
|---------|---------|----------|
|
||||
| Fixed ports in test compose | Parallel runs conflict | Use dynamic ports: `ports: ["5432"]` |
|
||||
| Persistent test volumes | Slow, stateful tests | Use `tmpfs` for ephemeral storage |
|
||||
| No healthchecks | Race conditions on startup | Add healthcheck + `depends_on: condition: service_healthy` |
|
||||
| Verbose test output | Context exhaustion | Use rich progress display, details only on failure |
|
||||
| Single test stage | Slow feedback, hard to debug | Separate unit → service → integration |
|
||||
| No branch isolation | Concurrent branches conflict | Use `TEST_INSTANCE_ID` from git branch |
|
||||
| Building all services on every change | Slow CI | Use matrix strategy, only build changed services |
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Component | Dev | Test | Prod |
|
||||
|-----------|-----|------|------|
|
||||
| Storage | Named volumes | tmpfs | External mounts |
|
||||
| Ports | Fixed (configurable) | Dynamic | Fixed |
|
||||
| Healthcheck interval | 30s | 5s | 30s |
|
||||
| Restart policy | None | None | unless-stopped |
|
||||
| Resource limits | None | None | Set limits |
|
||||
| Logging | Default | Default | json-file with rotation |
|
||||
|
||||
## Service Test Script Template
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scripts/test-service.sh
|
||||
set -e
|
||||
|
||||
SERVICE=$1
|
||||
TEST_INSTANCE_ID=$(./scripts/get-test-instance-id.sh)
|
||||
export TEST_INSTANCE_ID
|
||||
|
||||
# Start service-specific containers
|
||||
cd "services/${SERVICE}/deploy/test"
|
||||
docker compose up -d --build --wait
|
||||
|
||||
# Discover ports
|
||||
API_PORT=$(docker compose port api 8000 | cut -d: -f2)
|
||||
export TEST_API_URL="http://localhost:${API_PORT}"
|
||||
|
||||
# Run tests
|
||||
cd "../.."
|
||||
uv run pytest tests/integration/ --integration
|
||||
TEST_EXIT=$?
|
||||
|
||||
# Cleanup
|
||||
cd "deploy/test"
|
||||
docker compose down -v
|
||||
|
||||
exit $TEST_EXIT
|
||||
```
|
||||
Reference in New Issue
Block a user