mirror of
https://github.com/Xe138/AI-Trader.git
synced 2026-04-02 01:27:24 -04:00
Add Docker deployment design document
Comprehensive design for Docker containerization including: - Single monolithic container approach for dev/testing - docker-compose.yml with environment variable injection - Sequential startup via entrypoint.sh script - GitHub Actions workflow for automated image builds on release tags - Documentation strategy and implementation checklist
This commit is contained in:
491
docs/plans/2025-10-30-docker-deployment-design.md
Normal file
491
docs/plans/2025-10-30-docker-deployment-design.md
Normal file
@@ -0,0 +1,491 @@
|
||||
# Docker Deployment and CI/CD Design
|
||||
|
||||
**Date:** 2025-10-30
|
||||
**Status:** Approved
|
||||
**Target:** Development/local testing environment
|
||||
|
||||
## Overview
|
||||
|
||||
Package AI-Trader as a Docker container with docker-compose orchestration and automated image builds via GitHub Actions on release tags. Focus on simplicity and ease of use for researchers and developers.
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Primary Use Case:** Development and local testing
|
||||
- **Deployment Target:** Single monolithic container (all MCP services + trading agent)
|
||||
- **Secrets Management:** Environment variables (no mounted .env file)
|
||||
- **Data Strategy:** Fetch price data on container startup
|
||||
- **Container Registry:** GitHub Container Registry (ghcr.io)
|
||||
- **Trigger:** Build images automatically on release tag push (`v*` pattern)
|
||||
|
||||
## Architecture
|
||||
|
||||
### Components
|
||||
|
||||
1. **Dockerfile** - Builds Python 3.10 image with all dependencies
|
||||
2. **docker-compose.yml** - Orchestrates container with volume mounts and environment config
|
||||
3. **entrypoint.sh** - Sequential startup script (data fetch → MCP services → trading agent)
|
||||
4. **GitHub Actions Workflow** - Automated image build and push on release tags
|
||||
5. **.dockerignore** - Excludes unnecessary files from image
|
||||
6. **Documentation** - Docker usage guide and examples
|
||||
|
||||
### Execution Flow
|
||||
|
||||
```
|
||||
Container Start
|
||||
↓
|
||||
entrypoint.sh
|
||||
↓
|
||||
1. Fetch/merge price data (get_daily_price.py → merge_jsonl.py)
|
||||
↓
|
||||
2. Start MCP services in background (start_mcp_services.py)
|
||||
↓
|
||||
3. Wait 3 seconds for service stabilization
|
||||
↓
|
||||
4. Run trading agent (main.py with config)
|
||||
↓
|
||||
Container Exit → Cleanup MCP services
|
||||
```
|
||||
|
||||
## Detailed Design
|
||||
|
||||
### 1. Dockerfile
|
||||
|
||||
**Multi-stage build:**
|
||||
|
||||
```dockerfile
|
||||
# Base stage
|
||||
FROM python:3.10-slim as base
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install dependencies
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Application stage
|
||||
FROM base
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy application code
|
||||
COPY . .
|
||||
|
||||
# Create necessary directories
|
||||
RUN mkdir -p data logs data/agent_data
|
||||
|
||||
# Make entrypoint executable
|
||||
RUN chmod +x entrypoint.sh
|
||||
|
||||
# Expose MCP service ports
|
||||
EXPOSE 8000 8001 8002 8003
|
||||
|
||||
# Set Python to run unbuffered
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
# Use entrypoint script
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
CMD ["configs/default_config.json"]
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- `python:3.10-slim` base for smaller image size
|
||||
- Multi-stage for dependency caching
|
||||
- Non-root user NOT included (dev/testing focus, can add later)
|
||||
- Unbuffered Python output for real-time logs
|
||||
- Default config path with override support
|
||||
|
||||
### 2. docker-compose.yml
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
ai-trader:
|
||||
build: .
|
||||
container_name: ai-trader-app
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
- ./logs:/app/logs
|
||||
environment:
|
||||
- OPENAI_API_BASE=${OPENAI_API_BASE}
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY}
|
||||
- ALPHAADVANTAGE_API_KEY=${ALPHAADVANTAGE_API_KEY}
|
||||
- JINA_API_KEY=${JINA_API_KEY}
|
||||
- RUNTIME_ENV_PATH=/app/data/runtime_env.json
|
||||
- MATH_HTTP_PORT=${MATH_HTTP_PORT:-8000}
|
||||
- SEARCH_HTTP_PORT=${SEARCH_HTTP_PORT:-8001}
|
||||
- TRADE_HTTP_PORT=${TRADE_HTTP_PORT:-8002}
|
||||
- GETPRICE_HTTP_PORT=${GETPRICE_HTTP_PORT:-8003}
|
||||
- AGENT_MAX_STEP=${AGENT_MAX_STEP:-30}
|
||||
ports:
|
||||
- "8000:8000"
|
||||
- "8001:8001"
|
||||
- "8002:8002"
|
||||
- "8003:8003"
|
||||
- "8888:8888" # Optional: web dashboard
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- Volume mounts for data/logs persistence
|
||||
- Environment variables interpolated from `.env` file (Docker Compose reads automatically)
|
||||
- No `.env` file mounted into container (cleaner separation)
|
||||
- Default port values with override support
|
||||
- Restart policy for recovery
|
||||
|
||||
### 3. entrypoint.sh
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e # Exit on any error
|
||||
|
||||
echo "🚀 Starting AI-Trader..."
|
||||
|
||||
# Step 1: Data preparation
|
||||
echo "📊 Fetching and merging price data..."
|
||||
cd /app/data
|
||||
python get_daily_price.py
|
||||
python merge_jsonl.py
|
||||
cd /app
|
||||
|
||||
# Step 2: Start MCP services in background
|
||||
echo "🔧 Starting MCP services..."
|
||||
cd /app/agent_tools
|
||||
python start_mcp_services.py &
|
||||
MCP_PID=$!
|
||||
cd /app
|
||||
|
||||
# Step 3: Wait for services to initialize
|
||||
echo "⏳ Waiting for MCP services to start..."
|
||||
sleep 3
|
||||
|
||||
# Step 4: Run trading agent with config file
|
||||
echo "🤖 Starting trading agent..."
|
||||
CONFIG_FILE="${1:-configs/default_config.json}"
|
||||
python main.py "$CONFIG_FILE"
|
||||
|
||||
# Cleanup on exit
|
||||
trap "echo '🛑 Stopping MCP services...'; kill $MCP_PID 2>/dev/null" EXIT
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- Sequential execution with clear logging
|
||||
- MCP services run in background with PID capture
|
||||
- Trap ensures cleanup on container exit
|
||||
- Config file path as argument (defaults to `configs/default_config.json`)
|
||||
- Fail-fast with `set -e`
|
||||
|
||||
### 4. GitHub Actions Workflow
|
||||
|
||||
**File:** `.github/workflows/docker-release.yml`
|
||||
|
||||
```yaml
|
||||
name: Build and Push Docker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- 'v*' # Triggers on v1.0.0, v2.1.3, etc.
|
||||
workflow_dispatch: # Manual trigger option
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Extract version from tag
|
||||
id: meta
|
||||
run: |
|
||||
VERSION=${GITHUB_REF#refs/tags/v}
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Build and push Docker image
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
push: true
|
||||
tags: |
|
||||
ghcr.io/${{ github.repository_owner }}/ai-trader:${{ steps.meta.outputs.version }}
|
||||
ghcr.io/${{ github.repository_owner }}/ai-trader:latest
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- Triggers on `v*` tags (e.g., `git tag v1.0.0 && git push origin v1.0.0`)
|
||||
- Manual dispatch option for testing
|
||||
- Uses `GITHUB_TOKEN` (automatically provided, no secrets needed)
|
||||
- Builds with caching for faster builds
|
||||
- Tags both version and `latest`
|
||||
- Multi-platform support possible by adding `platforms: linux/amd64,linux/arm64`
|
||||
|
||||
### 5. .dockerignore
|
||||
|
||||
```
|
||||
# Version control
|
||||
.git/
|
||||
.gitignore
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
venv/
|
||||
env/
|
||||
ENV/
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Environment and secrets
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
|
||||
# Data files (fetched at runtime)
|
||||
data/*.json
|
||||
data/agent_data/
|
||||
data/merged.jsonl
|
||||
|
||||
# Logs
|
||||
logs/
|
||||
*.log
|
||||
|
||||
# Runtime state
|
||||
runtime_env.json
|
||||
|
||||
# Documentation (not needed in image)
|
||||
*.md
|
||||
docs/
|
||||
!README.md
|
||||
|
||||
# CI/CD
|
||||
.github/
|
||||
```
|
||||
|
||||
**Purpose:**
|
||||
- Reduces image size
|
||||
- Keeps secrets out of image
|
||||
- Excludes generated files
|
||||
- Keeps only necessary source code and scripts
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
### New File: docs/DOCKER.md
|
||||
|
||||
Create comprehensive Docker usage guide including:
|
||||
|
||||
1. **Quick Start**
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env with your API keys
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
2. **Configuration**
|
||||
- Required environment variables
|
||||
- Optional configuration overrides
|
||||
- Custom config file usage
|
||||
|
||||
3. **Usage Examples**
|
||||
```bash
|
||||
# Run with default config
|
||||
docker-compose up
|
||||
|
||||
# Run with custom config
|
||||
docker-compose run ai-trader configs/my_config.json
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Stop and clean up
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
4. **Data Persistence**
|
||||
- How volume mounts work
|
||||
- Where data is stored
|
||||
- How to backup/restore
|
||||
|
||||
5. **Troubleshooting**
|
||||
- MCP services not starting → Check logs, verify ports available
|
||||
- Missing API keys → Check .env file
|
||||
- Data fetch failures → API rate limits or invalid keys
|
||||
- Permission issues → Volume mount permissions
|
||||
|
||||
6. **Using Pre-built Images**
|
||||
```bash
|
||||
docker pull ghcr.io/hkuds/ai-trader:latest
|
||||
docker run --env-file .env -v $(pwd)/data:/app/data ghcr.io/hkuds/ai-trader:latest
|
||||
```
|
||||
|
||||
### Update .env.example
|
||||
|
||||
Add/clarify Docker-specific variables:
|
||||
|
||||
```bash
|
||||
# AI Model API Configuration
|
||||
OPENAI_API_BASE=https://your-openai-proxy.com/v1
|
||||
OPENAI_API_KEY=your_openai_key
|
||||
|
||||
# Data Source Configuration
|
||||
ALPHAADVANTAGE_API_KEY=your_alpha_vantage_key
|
||||
JINA_API_KEY=your_jina_api_key
|
||||
|
||||
# System Configuration (Docker defaults)
|
||||
RUNTIME_ENV_PATH=/app/data/runtime_env.json
|
||||
|
||||
# MCP Service Ports
|
||||
MATH_HTTP_PORT=8000
|
||||
SEARCH_HTTP_PORT=8001
|
||||
TRADE_HTTP_PORT=8002
|
||||
GETPRICE_HTTP_PORT=8003
|
||||
|
||||
# Agent Configuration
|
||||
AGENT_MAX_STEP=30
|
||||
```
|
||||
|
||||
### Update Main README.md
|
||||
|
||||
Add Docker section after "Quick Start":
|
||||
|
||||
```markdown
|
||||
## Docker Deployment
|
||||
|
||||
### Using Docker Compose (Recommended)
|
||||
|
||||
```bash
|
||||
# Setup environment
|
||||
cp .env.example .env
|
||||
# Edit .env with your API keys
|
||||
|
||||
# Run with docker-compose
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
### Using Pre-built Images
|
||||
|
||||
```bash
|
||||
# Pull latest image
|
||||
docker pull ghcr.io/hkuds/ai-trader:latest
|
||||
|
||||
# Run container
|
||||
docker run --env-file .env \
|
||||
-v $(pwd)/data:/app/data \
|
||||
-v $(pwd)/logs:/app/logs \
|
||||
ghcr.io/hkuds/ai-trader:latest
|
||||
```
|
||||
|
||||
See [docs/DOCKER.md](docs/DOCKER.md) for detailed Docker usage guide.
|
||||
```
|
||||
|
||||
## Release Process
|
||||
|
||||
### For Maintainers
|
||||
|
||||
1. **Prepare release:**
|
||||
```bash
|
||||
# Ensure main branch is ready
|
||||
git checkout main
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
2. **Create and push tag:**
|
||||
```bash
|
||||
git tag v1.0.0
|
||||
git push origin v1.0.0
|
||||
```
|
||||
|
||||
3. **GitHub Actions automatically:**
|
||||
- Builds Docker image
|
||||
- Tags with version and `latest`
|
||||
- Pushes to `ghcr.io/hkuds/ai-trader`
|
||||
|
||||
4. **Verify build:**
|
||||
- Check Actions tab for build status
|
||||
- Test pull: `docker pull ghcr.io/hkuds/ai-trader:v1.0.0`
|
||||
|
||||
5. **Optional: Create GitHub Release**
|
||||
- Add release notes
|
||||
- Include Docker pull command
|
||||
|
||||
### For Users
|
||||
|
||||
```bash
|
||||
# Pull specific version
|
||||
docker pull ghcr.io/hkuds/ai-trader:v1.0.0
|
||||
|
||||
# Or always get latest
|
||||
docker pull ghcr.io/hkuds/ai-trader:latest
|
||||
```
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] Create Dockerfile with multi-stage build
|
||||
- [ ] Create docker-compose.yml with volume mounts and environment config
|
||||
- [ ] Create entrypoint.sh with sequential startup logic
|
||||
- [ ] Create .dockerignore to exclude unnecessary files
|
||||
- [ ] Create .github/workflows/docker-release.yml for CI/CD
|
||||
- [ ] Create docs/DOCKER.md with comprehensive usage guide
|
||||
- [ ] Update .env.example with Docker-specific variables
|
||||
- [ ] Update main README.md with Docker deployment section
|
||||
- [ ] Test local build: `docker-compose build`
|
||||
- [ ] Test local run: `docker-compose up`
|
||||
- [ ] Test with custom config
|
||||
- [ ] Verify data persistence across container restarts
|
||||
- [ ] Test GitHub Actions workflow (create test tag)
|
||||
- [ ] Verify image pushed to ghcr.io
|
||||
- [ ] Test pulling and running pre-built image
|
||||
- [ ] Update CLAUDE.md with Docker commands
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Possible improvements for production use:
|
||||
|
||||
1. **Multi-container Architecture**
|
||||
- Separate containers for each MCP service
|
||||
- Better isolation and independent scaling
|
||||
- More complex orchestration
|
||||
|
||||
2. **Security Hardening**
|
||||
- Non-root user in container
|
||||
- Docker secrets for production
|
||||
- Read-only filesystem where possible
|
||||
|
||||
3. **Monitoring**
|
||||
- Health checks for MCP services
|
||||
- Prometheus metrics export
|
||||
- Logging aggregation
|
||||
|
||||
4. **Optimization**
|
||||
- Multi-platform builds (ARM64 support)
|
||||
- Smaller base image (alpine)
|
||||
- Layer caching optimization
|
||||
|
||||
5. **Development Tools**
|
||||
- docker-compose.dev.yml with hot reload
|
||||
- Debug container with additional tools
|
||||
- Integration test container
|
||||
|
||||
These are deferred to keep initial implementation simple and focused on development/testing use cases.
|
||||
Reference in New Issue
Block a user