feat: transform to REST API service with SQLite persistence (v0.3.0)

Major architecture transformation from batch-only to API service with database persistence for Windmill integration. ## REST API Implementation - POST /simulate/trigger - Start simulation jobs - GET /simulate/status/{job_id} - Monitor job progress - GET /results - Query results with filters (job_id, date, model) - GET /health - Service health checks ## Database Layer - SQLite persistence with 6 tables (jobs, job_details, positions, holdings, reasoning_logs, tool_usage) - Foreign key constraints with cascade deletes - Replaces JSONL file storage ## Backend Components - JobManager: Job lifecycle management with concurrency control - RuntimeConfigManager: Thread-safe isolated runtime configs - ModelDayExecutor: Single model-day execution engine - SimulationWorker: Date-sequential, model-parallel orchestration ## Testing - 102 unit and integration tests (85% coverage) - Database: 98% coverage - Job manager: 98% coverage - API endpoints: 81% coverage - Pydantic models: 100% coverage - TDD approach throughout ## Docker Deployment - Dual-mode: API server (persistent) + batch (one-time) - Health checks with 30s interval - Volume persistence for database and logs - Separate entrypoints for each mode ## Validation Tools - scripts/validate_docker_build.sh - Build validation - scripts/test_api_endpoints.sh - Complete API testing - scripts/test_batch_mode.sh - Batch mode validation - DOCKER_API.md - Deployment guide - TESTING_GUIDE.md - Testing procedures ## Configuration - API_PORT environment variable (default: 8080) - Backwards compatible with existing configs - FastAPI, uvicorn, pydantic>=2.0 dependencies Co-Authored-By: AI Assistant <noreply@example.com>
2026-04-01 17:17:24 -04:00 · 2025-10-31 11:47:10 -04:00
parent 5da02b4ba0
commit fb9583b374
45 changed files with 13775 additions and 18 deletions
--- a/TESTING_GUIDE.md
+++ b/TESTING_GUIDE.md
@@ -0,0 +1,513 @@
+# AI-Trader Testing & Validation Guide
+
+This guide provides step-by-step instructions for validating the AI-Trader Docker deployment.
+
+## Prerequisites
+
+- Docker Desktop installed and running
+- `.env` file configured with API keys
+- At least 2GB free disk space
+- Internet connection for initial price data download
+
+## Quick Start
+
+```bash
+# 1. Make scripts executable
+chmod +x scripts/*.sh
+
+# 2. Validate Docker build
+bash scripts/validate_docker_build.sh
+
+# 3. Test API endpoints
+bash scripts/test_api_endpoints.sh
+
+# 4. Test batch mode
+bash scripts/test_batch_mode.sh
+```
+
+---
+
+## Detailed Testing Procedures
+
+### Test 1: Docker Build Validation
+
+**Purpose:** Verify Docker image builds correctly and containers start
+
+**Command:**
+```bash
+bash scripts/validate_docker_build.sh
+```
+
+**What it tests:**
+- ✅ Docker and docker-compose installed
+- ✅ Docker daemon running
+- ✅ `.env` file exists and configured
+- ✅ Image builds successfully
+- ✅ Container starts in API mode
+- ✅ Health endpoint responds
+- ✅ No critical errors in logs
+
+**Expected output:**
+```
+==========================================
+AI-Trader Docker Build Validation
+==========================================
+
+Step 1: Checking prerequisites...
+✓ Docker is installed: Docker version 24.0.0
+✓ Docker daemon is running
+✓ docker-compose is installed
+
+Step 2: Checking environment configuration...
+✓ .env file exists
+✓ OPENAI_API_KEY is set
+✓ ALPHAADVANTAGE_API_KEY is set
+✓ JINA_API_KEY is set
+
+Step 3: Building Docker image...
+✓ Docker image built successfully
+
+Step 4: Verifying Docker image...
+✓ Image size: 850MB
+✓ Exposed ports: 8000/tcp 8001/tcp 8002/tcp 8003/tcp 8080/tcp 8888/tcp
+
+Step 5: Testing API mode startup...
+✓ Container started successfully
+✓ Container is running
+✓ No critical errors in logs
+
+Step 6: Testing health endpoint...
+✓ Health endpoint responding
+Health response: {"status":"healthy","database":"connected","timestamp":"..."}
+```
+
+**If it fails:**
+- Check Docker Desktop is running
+- Verify `.env` has all required keys
+- Check port 8080 is not already in use
+- Review logs: `docker logs ai-trader-api`
+
+---
+
+### Test 2: API Endpoint Testing
+
+**Purpose:** Validate all REST API endpoints work correctly
+
+**Command:**
+```bash
+# Ensure API is running first
+docker-compose up -d ai-trader-api
+
+# Run tests
+bash scripts/test_api_endpoints.sh
+```
+
+**What it tests:**
+- ✅ GET /health - Service health check
+- ✅ POST /simulate/trigger - Job creation
+- ✅ GET /simulate/status/{job_id} - Status tracking
+- ✅ Job completion monitoring
+- ✅ GET /results - Results retrieval
+- ✅ Query filtering (by date, model)
+- ✅ Concurrent job prevention
+- ✅ Error handling (invalid inputs)
+
+**Expected output:**
+```
+==========================================
+AI-Trader API Endpoint Testing
+==========================================
+
+✓ API is accessible
+
+Test 1: GET /health
+✓ Health check passed
+
+Test 2: POST /simulate/trigger
+✓ Simulation triggered successfully
+Job ID: 550e8400-e29b-41d4-a716-446655440000
+
+Test 3: GET /simulate/status/{job_id}
+✓ Job status retrieved
+Job Status: pending
+
+Test 4: Monitoring job progress
+[1/30] Status: running | Progress: {"completed":1,"failed":0,...}
+...
+✓ Job finished with status: completed
+
+Test 5: GET /results
+✓ Results retrieved
+Result count: 2
+
+Test 6: GET /results?date=...
+✓ Date-filtered results retrieved
+
+Test 7: GET /results?model=...
+✓ Model-filtered results retrieved
+
+Test 8: Concurrent job prevention
+✓ Concurrent job correctly rejected
+
+Test 9: Error handling
+✓ Invalid config path correctly rejected
+```
+
+**If it fails:**
+- Ensure container is running: `docker ps | grep ai-trader-api`
+- Check API logs: `docker logs ai-trader-api`
+- Verify port 8080 is accessible: `curl http://localhost:8080/health`
+- Check MCP services started: `docker exec ai-trader-api ps aux | grep python`
+
+---
+
+### Test 3: Batch Mode Testing
+
+**Purpose:** Verify one-time simulation execution works
+
+**Command:**
+```bash
+bash scripts/test_batch_mode.sh
+```
+
+**What it tests:**
+- ✅ Batch mode container starts
+- ✅ Simulation executes to completion
+- ✅ Exit code is 0 (success)
+- ✅ Position files created
+- ✅ Log files generated
+- ✅ Price data persists between runs
+
+**Expected output:**
+```
+==========================================
+AI-Trader Batch Mode Testing
+==========================================
+
+✓ Prerequisites OK
+Using config: configs/default_config.json
+
+Test 1: Building Docker image
+✓ Image built successfully
+
+Test 2: Running batch simulation
+🚀 Starting AI-Trader...
+✅ Environment variables validated
+📊 Fetching and merging price data...
+🔧 Starting MCP services...
+🤖 Starting trading agent...
+[Trading output...]
+
+Test 3: Checking exit status
+✓ Batch simulation completed successfully (exit code: 0)
+
+Test 4: Verifying output files
+✓ Found 1 position file(s)
+Sample position data: {...}
+✓ Found 3 log file(s)
+
+Test 5: Checking price data
+✓ Price data exists: 100 stocks
+
+Test 6: Testing data persistence
+✓ Second run completed successfully
+✓ Price data was reused
+```
+
+**If it fails:**
+- Check `.env` has valid API keys
+- Verify internet connection (for price data)
+- Check available disk space
+- Review batch logs: `docker logs ai-trader-batch`
+- Check data directory permissions
+
+---
+
+## Manual Testing Procedures
+
+### Test 1: API Health Check
+
+```bash
+# Start API
+docker-compose up -d ai-trader-api
+
+# Test health endpoint
+curl http://localhost:8080/health
+
+# Expected response:
+# {"status":"healthy","database":"connected","timestamp":"2025-01-16T10:00:00Z"}
+```
+
+### Test 2: Trigger Simulation
+
+```bash
+# Trigger job
+curl -X POST http://localhost:8080/simulate/trigger \
+  -H "Content-Type: application/json" \
+  -d '{
+    "config_path": "/app/configs/default_config.json",
+    "date_range": ["2025-01-16", "2025-01-17"],
+    "models": ["gpt-4"]
+  }'
+
+# Expected response:
+# {
+#   "job_id": "550e8400-e29b-41d4-a716-446655440000",
+#   "status": "pending",
+#   "total_model_days": 2,
+#   "message": "Simulation job ... created and started"
+# }
+
+# Save job_id for next steps
+JOB_ID="550e8400-e29b-41d4-a716-446655440000"
+```
+
+### Test 3: Monitor Job Progress
+
+```bash
+# Check status (repeat until completed)
+curl http://localhost:8080/simulate/status/$JOB_ID | jq '.'
+
+# Poll with watch
+watch -n 10 "curl -s http://localhost:8080/simulate/status/$JOB_ID | jq '.status, .progress'"
+```
+
+### Test 4: Retrieve Results
+
+```bash
+# Get all results for job
+curl "http://localhost:8080/results?job_id=$JOB_ID" | jq '.'
+
+# Filter by date
+curl "http://localhost:8080/results?date=2025-01-16" | jq '.'
+
+# Filter by model
+curl "http://localhost:8080/results?model=gpt-4" | jq '.'
+
+# Combine filters
+curl "http://localhost:8080/results?job_id=$JOB_ID&date=2025-01-16&model=gpt-4" | jq '.'
+```
+
+### Test 5: Volume Persistence
+
+```bash
+# Stop container
+docker-compose down
+
+# Verify data persists
+ls -lh data/jobs.db
+ls -R data/agent_data
+
+# Restart container
+docker-compose up -d ai-trader-api
+
+# Data should still be accessible via API
+curl http://localhost:8080/results | jq '.count'
+```
+
+---
+
+## Troubleshooting
+
+### Problem: Container won't start
+
+**Symptoms:**
+- `docker ps` shows no ai-trader-api container
+- Container exits immediately
+
+**Debug steps:**
+```bash
+# Check logs
+docker logs ai-trader-api
+
+# Common issues:
+# 1. Missing API keys in .env
+# 2. Port 8080 already in use
+# 3. Volume permission issues
+```
+
+**Solutions:**
+```bash
+# 1. Verify .env
+cat .env | grep -E "OPENAI_API_KEY|ALPHAADVANTAGE_API_KEY|JINA_API_KEY"
+
+# 2. Check port usage
+lsof -i :8080  # Linux/Mac
+netstat -ano | findstr :8080  # Windows
+
+# 3. Fix permissions
+chmod -R 755 data logs
+```
+
+### Problem: Health check fails
+
+**Symptoms:**
+- `curl http://localhost:8080/health` returns error
+- Container is running but API not responding
+
+**Debug steps:**
+```bash
+# Check if API process is running
+docker exec ai-trader-api ps aux | grep uvicorn
+
+# Check internal health
+docker exec ai-trader-api curl http://localhost:8080/health
+
+# Check logs for startup errors
+docker logs ai-trader-api | grep -i error
+```
+
+**Solutions:**
+```bash
+# If MCP services didn't start:
+docker exec ai-trader-api ps aux | grep python
+
+# If database issues:
+docker exec ai-trader-api ls -l /app/data/jobs.db
+
+# Restart container
+docker-compose restart ai-trader-api
+```
+
+### Problem: Job stays in "pending" status
+
+**Symptoms:**
+- Job triggered but never progresses
+- Status remains "pending" indefinitely
+
+**Debug steps:**
+```bash
+# Check worker logs
+docker logs ai-trader-api | grep -i "worker\|simulation"
+
+# Check database
+docker exec ai-trader-api sqlite3 /app/data/jobs.db "SELECT * FROM job_details;"
+
+# Check if MCP services are accessible
+docker exec ai-trader-api curl http://localhost:8000/health
+```
+
+**Solutions:**
+```bash
+# Restart container (jobs resume automatically)
+docker-compose restart ai-trader-api
+
+# Check specific job status
+curl http://localhost:8080/simulate/status/$JOB_ID | jq '.details'
+```
+
+### Problem: Tests timeout
+
+**Symptoms:**
+- `test_api_endpoints.sh` hangs during job monitoring
+- Jobs take longer than expected
+
+**Solutions:**
+```bash
+# Increase poll timeout in test script
+# Edit: MAX_POLLS=60  # Increase from 30
+
+# Or monitor job manually
+watch -n 30 "curl -s http://localhost:8080/simulate/status/$JOB_ID | jq '.status, .progress'"
+
+# Check agent logs for slowness
+docker logs ai-trader-api | tail -100
+```
+
+---
+
+## Performance Benchmarks
+
+### Expected Execution Times
+
+**Docker Build:**
+- First build: 5-10 minutes
+- Subsequent builds: 1-2 minutes (with cache)
+
+**API Startup:**
+- Container start: 5-10 seconds
+- Health check ready: 15-20 seconds (including MCP services)
+
+**Single Model-Day Simulation:**
+- With existing price data: 2-5 minutes
+- First run (fetching price data): 10-15 minutes
+
+**Complete 2-Date, 2-Model Job:**
+- Expected duration: 10-20 minutes
+- Depends on AI model response times
+
+---
+
+## Continuous Monitoring
+
+### Health Check Monitoring
+
+```bash
+# Add to cron for continuous monitoring
+*/5 * * * * curl -f http://localhost:8080/health || echo "API down" | mail -s "AI-Trader Alert" admin@example.com
+```
+
+### Log Rotation
+
+```bash
+# Docker handles log rotation, but monitor size:
+docker logs ai-trader-api --tail 100
+
+# Clear old logs if needed:
+docker logs ai-trader-api > /dev/null 2>&1
+```
+
+### Database Size
+
+```bash
+# Monitor database growth
+docker exec ai-trader-api du -h /app/data/jobs.db
+
+# Vacuum periodically
+docker exec ai-trader-api sqlite3 /app/data/jobs.db "VACUUM;"
+```
+
+---
+
+## Success Criteria
+
+### Validation Complete When:
+
+- ✅ All 3 test scripts pass without errors
+- ✅ Health endpoint returns "healthy" status
+- ✅ Can trigger and complete simulation job
+- ✅ Results are retrievable via API
+- ✅ Data persists after container restart
+- ✅ Batch mode completes successfully
+- ✅ No critical errors in logs
+
+### Ready for Production When:
+
+- ✅ All validation tests pass
+- ✅ Performance meets expectations
+- ✅ Monitoring is configured
+- ✅ Backup strategy is in place
+- ✅ Documentation is reviewed
+- ✅ Team is trained on operations
+
+---
+
+## Next Steps After Validation
+
+1. **Set up monitoring** - Configure health check alerts
+2. **Configure backups** - Backup `/app/data` regularly
+3. **Document operations** - Create runbook for team
+4. **Set up CI/CD** - Automate testing and deployment
+5. **Integrate with Windmill** - Connect workflows to API
+6. **Scale if needed** - Deploy multiple instances with load balancer
+
+---
+
+## Support
+
+For issues not covered in this guide:
+
+1. Check `DOCKER_API.md` for detailed API documentation
+2. Review container logs: `docker logs ai-trader-api`
+3. Check database: `docker exec ai-trader-api sqlite3 /app/data/jobs.db ".tables"`
+4. Open issue on GitHub with logs and error messages