AI-Trader/docs/plans/2025-10-30-docker-deployment-design.md

Docker Deployment and CI/CD Design

Date: 2025-10-30 Status: Approved Target: Development/local testing environment

Overview

Package AI-Trader as a Docker container with docker-compose orchestration and automated image builds via GitHub Actions on release tags. Focus on simplicity and ease of use for researchers and developers.

Requirements

• Primary Use Case: Development and local testing
• Deployment Target: Single monolithic container (all MCP services + trading agent)
• Secrets Management: Environment variables (no mounted .env file)
• Data Strategy: Fetch price data on container startup
• Container Registry: GitHub Container Registry (ghcr.io)
• Trigger: Build images automatically on release tag push (v* pattern)

Architecture

Components

1. Dockerfile - Builds Python 3.10 image with all dependencies
2. docker-compose.yml - Orchestrates container with volume mounts and environment config
3. entrypoint.sh - Sequential startup script (data fetch → MCP services → trading agent)
4. GitHub Actions Workflow - Automated image build and push on release tags
5. .dockerignore - Excludes unnecessary files from image
6. Documentation - Docker usage guide and examples

Execution Flow

Container Start
    ↓
entrypoint.sh
    ↓
1. Fetch/merge price data (get_daily_price.py → merge_jsonl.py)
    ↓
2. Start MCP services in background (start_mcp_services.py)
    ↓
3. Wait 3 seconds for service stabilization
    ↓
4. Run trading agent (main.py with config)
    ↓
Container Exit → Cleanup MCP services

Detailed Design

1. Dockerfile

Multi-stage build:

# Base stage
FROM python:3.10-slim as base

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Application stage
FROM base

WORKDIR /app

# Copy application code
COPY . .

# Create necessary directories
RUN mkdir -p data logs data/agent_data

# Make entrypoint executable
RUN chmod +x entrypoint.sh

# Expose MCP service ports
EXPOSE 8000 8001 8002 8003

# Set Python to run unbuffered
ENV PYTHONUNBUFFERED=1

# Use entrypoint script
ENTRYPOINT ["./entrypoint.sh"]
CMD ["configs/default_config.json"]

Key Features:

• python:3.10-slim base for smaller image size
• Multi-stage for dependency caching
• Non-root user NOT included (dev/testing focus, can add later)
• Unbuffered Python output for real-time logs
• Default config path with override support

2. docker-compose.yml

version: '3.8'

services:
  ai-trader:
    build: .
    container_name: ai-trader-app
    volumes:
      - ./data:/app/data
      - ./logs:/app/logs
    environment:
      - OPENAI_API_BASE=${OPENAI_API_BASE}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ALPHAADVANTAGE_API_KEY=${ALPHAADVANTAGE_API_KEY}
      - JINA_API_KEY=${JINA_API_KEY}
      - RUNTIME_ENV_PATH=/app/data/runtime_env.json
      - MATH_HTTP_PORT=${MATH_HTTP_PORT:-8000}
      - SEARCH_HTTP_PORT=${SEARCH_HTTP_PORT:-8001}
      - TRADE_HTTP_PORT=${TRADE_HTTP_PORT:-8002}
      - GETPRICE_HTTP_PORT=${GETPRICE_HTTP_PORT:-8003}
      - AGENT_MAX_STEP=${AGENT_MAX_STEP:-30}
    ports:
      - "8000:8000"
      - "8001:8001"
      - "8002:8002"
      - "8003:8003"
      - "8888:8888"  # Optional: web dashboard
    restart: unless-stopped

Key Features:

• Volume mounts for data/logs persistence
• Environment variables interpolated from .env file (Docker Compose reads automatically)
• No .env file mounted into container (cleaner separation)
• Default port values with override support
• Restart policy for recovery

3. entrypoint.sh

#!/bin/bash
set -e  # Exit on any error

echo "🚀 Starting AI-Trader..."

# Step 1: Data preparation
echo "📊 Fetching and merging price data..."
cd /app/data
python get_daily_price.py
python merge_jsonl.py
cd /app

# Step 2: Start MCP services in background
echo "🔧 Starting MCP services..."
cd /app/agent_tools
python start_mcp_services.py &
MCP_PID=$!
cd /app

# Step 3: Wait for services to initialize
echo "⏳ Waiting for MCP services to start..."
sleep 3

# Step 4: Run trading agent with config file
echo "🤖 Starting trading agent..."
CONFIG_FILE="${1:-configs/default_config.json}"
python main.py "$CONFIG_FILE"

# Cleanup on exit
trap "echo '🛑 Stopping MCP services...'; kill $MCP_PID 2>/dev/null" EXIT

Key Features:

• Sequential execution with clear logging
• MCP services run in background with PID capture
• Trap ensures cleanup on container exit
• Config file path as argument (defaults to configs/default_config.json)
• Fail-fast with set -e

4. GitHub Actions Workflow

File: .github/workflows/docker-release.yml

name: Build and Push Docker Image

on:
  push:
    tags:
      - 'v*'  # Triggers on v1.0.0, v2.1.3, etc.
  workflow_dispatch:  # Manual trigger option

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract version from tag
        id: meta
        run: |
          VERSION=${GITHUB_REF#refs/tags/v}
          echo "version=$VERSION" >> $GITHUB_OUTPUT

      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            ghcr.io/${{ github.repository_owner }}/ai-trader:${{ steps.meta.outputs.version }}
            ghcr.io/${{ github.repository_owner }}/ai-trader:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

Key Features:

• Triggers on v* tags (e.g., git tag v1.0.0 && git push origin v1.0.0)
• Manual dispatch option for testing
• Uses GITHUB_TOKEN (automatically provided, no secrets needed)
• Builds with caching for faster builds
• Tags both version and latest
• Multi-platform support possible by adding platforms: linux/amd64,linux/arm64

5. .dockerignore

# Version control
.git/
.gitignore

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
venv/
env/
ENV/

# IDE
.vscode/
.idea/
*.swp
*.swo

# Environment and secrets
.env
.env.*
!.env.example

# Data files (fetched at runtime)
data/*.json
data/agent_data/
data/merged.jsonl

# Logs
logs/
*.log

# Runtime state
runtime_env.json

# Documentation (not needed in image)
*.md
docs/
!README.md

# CI/CD
.github/

Purpose:

• Reduces image size
• Keeps secrets out of image
• Excludes generated files
• Keeps only necessary source code and scripts

Documentation Updates

New File: docs/DOCKER.md

Create comprehensive Docker usage guide including:

1. Quick Start

   cp .env.example .env
   # Edit .env with your API keys
   docker-compose up
   

2. Configuration

   • Required environment variables
   • Optional configuration overrides
   • Custom config file usage

3. Usage Examples

   # Run with default config
   docker-compose up
   
   # Run with custom config
   docker-compose run ai-trader configs/my_config.json
   
   # View logs
   docker-compose logs -f
   
   # Stop and clean up
   docker-compose down
   

4. Data Persistence

   • How volume mounts work
   • Where data is stored
   • How to backup/restore

5. Troubleshooting

   • MCP services not starting → Check logs, verify ports available
   • Missing API keys → Check .env file
   • Data fetch failures → API rate limits or invalid keys
   • Permission issues → Volume mount permissions

6. Using Pre-built Images

   docker pull ghcr.io/hkuds/ai-trader:latest
   docker run --env-file .env -v $(pwd)/data:/app/data ghcr.io/hkuds/ai-trader:latest
   

Update .env.example

Add/clarify Docker-specific variables:

# AI Model API Configuration
OPENAI_API_BASE=https://your-openai-proxy.com/v1
OPENAI_API_KEY=your_openai_key

# Data Source Configuration
ALPHAADVANTAGE_API_KEY=your_alpha_vantage_key
JINA_API_KEY=your_jina_api_key

# System Configuration (Docker defaults)
RUNTIME_ENV_PATH=/app/data/runtime_env.json

# MCP Service Ports
MATH_HTTP_PORT=8000
SEARCH_HTTP_PORT=8001
TRADE_HTTP_PORT=8002
GETPRICE_HTTP_PORT=8003

# Agent Configuration
AGENT_MAX_STEP=30

Update Main README.md

Add Docker section after "Quick Start":

## Docker Deployment

### Using Docker Compose (Recommended)

```bash
# Setup environment
cp .env.example .env
# Edit .env with your API keys

# Run with docker-compose
docker-compose up

Using Pre-built Images

# Pull latest image
docker pull ghcr.io/hkuds/ai-trader:latest

# Run container
docker run --env-file .env \
  -v $(pwd)/data:/app/data \
  -v $(pwd)/logs:/app/logs \
  ghcr.io/hkuds/ai-trader:latest

See docs/DOCKER.md for detailed Docker usage guide.

## Release Process

### For Maintainers

1. **Prepare release:**
   ```bash
   # Ensure main branch is ready
   git checkout main
   git pull origin main

2. Create and push tag:

   git tag v1.0.0
   git push origin v1.0.0
   

3. GitHub Actions automatically:

   • Builds Docker image
   • Tags with version and latest
   • Pushes to ghcr.io/hkuds/ai-trader

4. Verify build:

   • Check Actions tab for build status
   • Test pull: docker pull ghcr.io/hkuds/ai-trader:v1.0.0

5. Optional: Create GitHub Release

   • Add release notes
   • Include Docker pull command

For Users

# Pull specific version
docker pull ghcr.io/hkuds/ai-trader:v1.0.0

# Or always get latest
docker pull ghcr.io/hkuds/ai-trader:latest

Implementation Checklist

• Create Dockerfile with multi-stage build
• Create docker-compose.yml with volume mounts and environment config
• Create entrypoint.sh with sequential startup logic
• Create .dockerignore to exclude unnecessary files
• Create .github/workflows/docker-release.yml for CI/CD
• Create docs/DOCKER.md with comprehensive usage guide
• Update .env.example with Docker-specific variables
• Update main README.md with Docker deployment section
• Test local build: docker-compose build
• Test local run: docker-compose up
• Test with custom config
• Verify data persistence across container restarts
• Test GitHub Actions workflow (create test tag)
• Verify image pushed to ghcr.io
• Test pulling and running pre-built image
• Update CLAUDE.md with Docker commands

Future Enhancements

Possible improvements for production use:

1. Multi-container Architecture

   • Separate containers for each MCP service
   • Better isolation and independent scaling
   • More complex orchestration

2. Security Hardening

   • Non-root user in container
   • Docker secrets for production
   • Read-only filesystem where possible

3. Monitoring

   • Health checks for MCP services
   • Prometheus metrics export
   • Logging aggregation

4. Optimization

   • Multi-platform builds (ARM64 support)
   • Smaller base image (alpine)
   • Layer caching optimization

5. Development Tools

   • docker-compose.dev.yml with hot reload
   • Debug container with additional tools
   • Integration test container

These are deferred to keep initial implementation simple and focused on development/testing use cases.