AI-Trader/CHANGELOG.md

Changelog

All notable changes to the AI-Trader-Server project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Fixed

• Critical: Fixed position tracking bugs causing cash reset and positions lost over weekends
  • Removed redundant ModelDayExecutor._write_results_to_db() that created corrupt records with cash=0 and holdings=[]
  • Fixed profit calculation to compare against start-of-day portfolio value instead of previous day's final value
  • Positions now correctly carry over between trading days and across weekends
  • Profit/loss calculations now accurately reflect trading gains/losses without treating trades as losses

Changed

• Position tracking now exclusively handled by trade tools (buy(), sell()) and add_no_trade_record_to_db()
• Daily profit calculation compares to start-of-day (action_id=0) portfolio value for accurate P&L tracking

0.3.0 - 2025-11-03

Added - Development & Testing Features

• Development Mode - Mock AI provider for cost-free testing
  • DEPLOYMENT_MODE=DEV enables mock AI responses with deterministic stock rotation
  • Isolated dev database (trading_dev.db) separate from production data
  • PRESERVE_DEV_DATA=true option to prevent dev database reset on startup
  • No AI API costs during development and testing
  • All API responses include deployment_mode field
  • Startup warning displayed when running in DEV mode
• Config Override System - Docker configuration merging
  • Place custom configs in user-configs/ directory
  • Startup merges user config with default config
  • Comprehensive validation with clear error messages
  • Volume mount: ./user-configs:/app/user-configs

Added - Enhanced API Features

• Async Price Download - Non-blocking data preparation
  • POST /simulate/trigger no longer blocks on price downloads
  • New job status: downloading_data during data preparation
  • Warnings field in status response for download issues
  • Better user experience for large date ranges
• Resume Mode - Idempotent simulation execution
  • Jobs automatically skip already-completed model-days
  • Safe to re-run jobs without duplicating work
  • status="skipped" for already-completed executions
  • Error-free job completion when partial results exist
• Reasoning Logs API - Access AI decision-making history
  • GET /reasoning endpoint for querying reasoning logs
  • Filter by job_id, model_name, date, include_full_conversation
  • Includes conversation history and tool usage
  • Database-only storage (no JSONL files)
  • AI-powered summary generation for reasoning sessions
• Job Skip Status - Enhanced job status tracking
  • New status: skipped for already-completed model-days
  • Better differentiation between pending, running, and skipped
  • Accurate job completion detection

Added - Price Data Management & On-Demand Downloads

• SQLite Price Data Storage - Replaced JSONL files with relational database
  • price_data table for OHLCV data (replaces merged.jsonl)
  • price_data_coverage table for tracking downloaded date ranges
  • simulation_runs table for soft-delete position tracking
  • Comprehensive indexes for query performance
• On-Demand Price Data Downloads - Automatic gap filling via Alpha Vantage
  • Priority-based download strategy (maximize date completion)
  • Graceful rate limit handling (no pre-configured limits needed)
  • Smart coverage gap detection
  • Configurable via AUTO_DOWNLOAD_PRICE_DATA (default: true)
• Date Range API - Simplified date specification
  • Single date: {"start_date": "2025-01-20"}
  • Date range: {"start_date": "2025-01-20", "end_date": "2025-01-24"}
  • Automatic validation (chronological order, max range, not future)
  • Configurable max days via MAX_SIMULATION_DAYS (default: 30)
• Migration Tooling - Script to import existing merged.jsonl data
  • scripts/migrate_price_data.py for one-time data migration
  • Automatic coverage tracking during migration

Added - API Service Transformation

• REST API Service - Complete FastAPI implementation for external orchestration
  • POST /simulate/trigger - Trigger simulation jobs with config, date range, and models
  • GET /simulate/status/{job_id} - Query job progress and execution details
  • GET /results - Retrieve simulation results with filtering (job_id, date, model)
  • GET /health - Service health check with database connectivity verification
• SQLite Database - Complete persistence layer replacing JSONL files
  • Jobs table - Job metadata and lifecycle tracking
  • Job details table - Per model-day execution status
  • Positions table - Trading position records with P&L
  • Holdings table - Portfolio holdings breakdown
  • Reasoning logs table - AI decision reasoning history
  • Tool usage table - MCP tool usage statistics
• Backend Components
  • JobManager - Job lifecycle management with concurrent job prevention
  • RuntimeConfigManager - Isolated runtime configs for thread-safe execution
  • ModelDayExecutor - Single model-day execution engine
  • SimulationWorker - Job orchestration with date-sequential, model-parallel execution
• Comprehensive Test Suite
  • 175 unit and integration tests
  • 19 database tests (98% coverage)
  • 23 job manager tests (98% coverage)
  • 10 model executor tests (84% coverage)
  • 20 API endpoint tests (81% coverage)
  • 20 Pydantic model tests (100% coverage)
  • 10 runtime manager tests (89% coverage)
  • 22 date utilities tests (100% coverage)
  • 33 price data manager tests (85% coverage)
  • 10 on-demand download integration tests
  • 8 existing integration tests
• Docker Deployment - Persistent REST API service
  • API-only deployment (batch mode removed for simplicity)
  • Single docker-compose service (ai-trader-server)
  • Health check configuration (30s interval, 3 retries)
  • Volume persistence for SQLite database and logs
  • Configurable API_PORT for flexible deployment
  • System dependencies (curl, procps) for health checks and debugging
• Validation & Testing Tools
  • scripts/validate_docker_build.sh - Docker build and startup validation with port awareness
  • scripts/test_api_endpoints.sh - Complete API endpoint testing suite with port awareness
  • TESTING_GUIDE.md - Comprehensive testing procedures and troubleshooting (including port conflicts)
• Documentation
  • DOCKER_API.md - API deployment guide with examples
  • TESTING_GUIDE.md - Validation procedures and troubleshooting
  • API endpoint documentation with request/response examples
  • Windmill integration patterns and examples

Changed

• Project Rebrand - AI-Trader renamed to AI-Trader-Server
  • Updated all documentation for new project name
  • Updated Docker images to ghcr.io/xe138/ai-trader-server
  • Updated GitHub Actions workflows
  • Updated README, CHANGELOG, and all user guides
• Architecture - Transformed from batch-only to API-first service with database persistence
• Data Storage - Migrated from JSONL files to SQLite relational database
  • Price data now stored in price_data table instead of merged.jsonl
  • Tools/price_tools.py updated to query database
  • Position data fully migrated to database-only storage (removed JSONL dependencies)
  • Trade tools now read/write from database tables with lazy context injection
• Deployment - Simplified to single API-only Docker service (REST API is new in v0.3.0)
• Logging - Removed duplicate MCP service log files for cleaner output
• Configuration - Simplified environment variable configuration
  • Added: DEPLOYMENT_MODE (PROD/DEV) for environment control
  • Added: PRESERVE_DEV_DATA (default: false) to keep dev data between runs
  • Added: AUTO_DOWNLOAD_PRICE_DATA (default: true) - Enable on-demand downloads
  • Added: MAX_SIMULATION_DAYS (default: 30) - Maximum date range size
  • Added: API_PORT for host port mapping (default: 8080, customizable for port conflicts)
  • Removed: RUNTIME_ENV_PATH (API dynamically manages runtime configs)
  • Removed: MCP service ports (MATH_HTTP_PORT, SEARCH_HTTP_PORT, TRADE_HTTP_PORT, GETPRICE_HTTP_PORT)
  • Removed: WEB_HTTP_PORT (web UI not implemented)
  • MCP services use fixed internal ports (8000-8003) and are no longer exposed to host
  • Container always uses port 8080 internally for API
  • Only API port (8080) is exposed to host
  • Reduces configuration complexity and attack surface
• Requirements - Added fastapi>=0.120.0, uvicorn[standard]>=0.27.0, pydantic>=2.0.0
• Docker Compose - Single service (ai-trader-server) instead of dual-mode
• Dockerfile - Added system dependencies (curl, procps) and port 8080 exposure
• .env.example - Simplified configuration with only essential variables
• Entrypoint - Unified entrypoint.sh with proper signal handling (exec uvicorn)

Technical Implementation

• Test-Driven Development - All components written with tests first
• Mock-based Testing - Avoid heavy dependencies in unit tests
• Pydantic V2 - Type-safe request/response validation
• Foreign Key Constraints - Database referential integrity with cascade deletes
• Thread-safe Execution - Isolated runtime configs per model-day
• Background Job Execution - ThreadPoolExecutor for parallel model execution
• Automatic Status Transitions - Job status updates based on model-day completion

Performance & Quality

• Test Suite - 175 tests, all passing
  • Unit tests: 155 tests
  • Integration tests: 18 tests
  • API tests: 20+ tests
• Code Coverage - High coverage for new modules
  • Date utilities: 100%
  • Price data manager: 85%
  • Database layer: 98%
  • Job manager: 98%
  • Pydantic models: 100%
  • Runtime manager: 89%
  • Model executor: 84%
  • FastAPI app: 81%
• Test Execution - Fast test suite (~12 seconds for full suite)

Integration Ready

• Windmill.dev - HTTP-based integration with polling support
• External Orchestration - RESTful API for workflow automation
• Monitoring - Health checks and status tracking
• Persistence - SQLite database survives container restarts

Fixed

• Context Injection - Runtime parameters correctly injected into MCP tools
  • ContextInjector always overrides AI-provided parameters (defense-in-depth)
  • Hidden context parameters from AI tool schema to prevent hallucination
  • Resolved database locking issues with concurrent tool calls
  • Proper async handling of tool reloading after context injection
• Simulation Re-runs - Prevent duplicate execution of completed model-days
  • Fixed job hanging when re-running partially completed simulations
  • _execute_date() now skips already-completed model-days
  • Job completion status correctly reflects skipped items
• Agent Initialization - Correct parameter passing in API mode
  • Fixed BaseAgent initialization parameters in ModelDayExecutor
  • Resolved async execution and position storage issues
• Database Reliability - Various improvements for concurrent access
  • Fixed column existence checks before creating indexes
  • Proper database path resolution in dev mode (prevents recursive _dev suffix)
  • Module-level database initialization for uvicorn reliability
  • Fixed database locking during concurrent writes
  • Improved error handling in buy/sell functions
• Configuration - Improved config handling
  • Use enabled field from config to determine which models run
  • Use config models when empty models list provided
  • Correct handling of merged runtime configs in containers
  • Proper get_db_path() usage to pass base database path
• Docker - Various deployment improvements
  • Removed non-existent data scripts from Dockerfile
  • Proper respect for dev mode in entrypoint database initialization
  • Correct closure usage to capture db_path in lifespan context manager

Breaking Changes

• Batch Mode Removed - All simulations now run through REST API
  • v0.2.0 used sequential batch execution via Docker entrypoint
  • v0.3.0 introduces REST API for external orchestration
  • Migration: Use POST /simulate/trigger endpoint instead of direct script execution
• Data Storage Format Changed - Price data moved from JSONL to SQLite
  • Run python scripts/migrate_price_data.py to migrate existing merged.jsonl data
  • merged.jsonl no longer used (replaced by price_data table)
  • Automatic on-demand downloads eliminate need for manual data fetching
• Configuration Variables Changed
  • Added: DEPLOYMENT_MODE, PRESERVE_DEV_DATA, AUTO_DOWNLOAD_PRICE_DATA, MAX_SIMULATION_DAYS, API_PORT
  • Removed: RUNTIME_ENV_PATH, MCP service ports, WEB_HTTP_PORT
  • MCP services now use fixed internal ports (not exposed to host)

0.2.0 - 2025-10-31

Added

• Complete Docker deployment support with containerization
• Docker Compose orchestration for easy local deployment
• Multi-stage Dockerfile with Python 3.10-slim base image
• Automated CI/CD pipeline via GitHub Actions for release builds
• Automatic draft release creation with version tagging
• Docker images published to GitHub Container Registry (ghcr.io)
• Comprehensive Docker documentation (docs/DOCKER.md)
• Release process documentation (docs/RELEASING.md)
• Data cache reuse design documentation (docs/DESIGN_DATA_CACHE_REUSE.md)
• CLAUDE.md repository guidance for development
• Docker deployment section in main README
• Environment variable configuration via docker-compose
• Sequential startup script (entrypoint.sh) for data fetch, MCP services, and trading agent
• Volume mounts for data and logs persistence
• Pre-built image support from ghcr.io/xe138/ai-trader-server
• Configurable volume path for persistent data
• Configurable web interface host port
• Automated merged.jsonl creation during price fetching
• API key registration URLs in .env.example

Changed

• Updated .env.example with Docker-specific configuration, API key URLs, and paths
• Updated .gitignore to exclude git worktrees directory
• Removed deprecated version tag from docker-compose.yml
• Updated repository URLs to Xe138/AI-Trader-Server fork
• Docker Compose now uses pre-built image by default
• Simplified Docker config file selection with convention over configuration
• Fixed internal ports with configurable host ports
• Separated data scripts from volume mount directory
• Reduced log flooding during data fetch
• OPENAI_API_BASE can now be left empty in configuration

Fixed

• Docker Compose configuration now follows modern best practices (version-less)
• Prevent restart loop on missing API keys with proper validation
• Docker tag generation now converts repository owner to lowercase
• Validate GITHUB_REF is a tag in docker-release workflow
• Correct Dockerfile FROM AS casing
• Module import errors for MCP services resolved with PYTHONPATH
• Prevent price data overwrite on container restart
• Merge script now writes to current directory for volume compatibility

0.1.0 - Initial Release

Added

• AI trading competition platform for NASDAQ 100 stocks
• Support for multiple AI models (GPT, Claude, Qwen, DeepSeek, Gemini)
• MCP (Model Context Protocol) toolchain integration
  • Mathematical calculation tools
  • Market intelligence search via Jina AI
  • Trading execution tools
  • Price query tools
• Historical replay architecture with anti-look-ahead controls
• Alpha Vantage API integration for price data
• Autonomous AI decision-making with zero human intervention
• Real-time performance analytics and leaderboard
• Position tracking and trading logs
• Web-based performance dashboard
• Complete NASDAQ 100 stock universe support
• Initial capital: $10,000 per AI model
• Configurable date range for backtesting
• Multi-model concurrent trading support
• Automatic data fetching and merging
• Comprehensive README with quick start guide

Technical Details

• Python 3.10+ support
• LangChain framework integration
• FastMCP for MCP service implementation
• JSONL format for position and log storage
• Weekday-only trading simulation
• Configurable agent parameters (max_steps, max_retries, initial_cash)

---

Release Notes Template

For future releases, use this template:

## [X.Y.Z] - YYYY-MM-DD

### Added
- New features

### Changed
- Changes to existing functionality

### Deprecated
- Soon-to-be removed features

### Removed
- Removed features

### Fixed
- Bug fixes

### Security
- Security improvements

---