# Changelog

All notable changes to the AI-Trader-Server project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.3.0] - 2025-10-31

### Added - Price Data Management & On-Demand Downloads
- **SQLite Price Data Storage** - Replaced JSONL files with relational database
  - `price_data` table for OHLCV data (replaces merged.jsonl)
  - `price_data_coverage` table for tracking downloaded date ranges
  - `simulation_runs` table for soft-delete position tracking
  - Comprehensive indexes for query performance
- **On-Demand Price Data Downloads** - Automatic gap filling via Alpha Vantage
  - Priority-based download strategy (maximize date completion)
  - Graceful rate limit handling (no pre-configured limits needed)
  - Smart coverage gap detection
  - Configurable via `AUTO_DOWNLOAD_PRICE_DATA` (default: true)
- **Date Range API** - Simplified date specification
  - Single date: `{"start_date": "2025-01-20"}`
  - Date range: `{"start_date": "2025-01-20", "end_date": "2025-01-24"}`
  - Automatic validation (chronological order, max range, not future)
  - Configurable max days via `MAX_SIMULATION_DAYS` (default: 30)
- **Migration Tooling** - Script to import existing merged.jsonl data
  - `scripts/migrate_price_data.py` for one-time data migration
  - Automatic coverage tracking during migration

### Added - API Service Transformation
- **REST API Service** - Complete FastAPI implementation for external orchestration
  - `POST /simulate/trigger` - Trigger simulation jobs with config, date range, and models
  - `GET /simulate/status/{job_id}` - Query job progress and execution details
  - `GET /results` - Retrieve simulation results with filtering (job_id, date, model)
  - `GET /health` - Service health check with database connectivity verification
- **SQLite Database** - Complete persistence layer replacing JSONL files
  - Jobs table - Job metadata and lifecycle tracking
  - Job details table - Per model-day execution status
  - Positions table - Trading position records with P&L
  - Holdings table - Portfolio holdings breakdown
  - Reasoning logs table - AI decision reasoning history
  - Tool usage table - MCP tool usage statistics
- **Backend Components**
  - JobManager - Job lifecycle management with concurrent job prevention
  - RuntimeConfigManager - Isolated runtime configs for thread-safe execution
  - ModelDayExecutor - Single model-day execution engine
  - SimulationWorker - Job orchestration with date-sequential, model-parallel execution
- **Comprehensive Test Suite**
  - 175 unit and integration tests
  - 19 database tests (98% coverage)
  - 23 job manager tests (98% coverage)
  - 10 model executor tests (84% coverage)
  - 20 API endpoint tests (81% coverage)
  - 20 Pydantic model tests (100% coverage)
  - 10 runtime manager tests (89% coverage)
  - 22 date utilities tests (100% coverage)
  - 33 price data manager tests (85% coverage)
  - 10 on-demand download integration tests
  - 8 existing integration tests
- **Docker Deployment** - Persistent REST API service
  - API-only deployment (batch mode removed for simplicity)
  - Single docker-compose service (ai-trader-server)
  - Health check configuration (30s interval, 3 retries)
  - Volume persistence for SQLite database and logs
  - Configurable API_PORT for flexible deployment
  - System dependencies (curl, procps) for health checks and debugging
- **Validation & Testing Tools**
  - `scripts/validate_docker_build.sh` - Docker build and startup validation with port awareness
  - `scripts/test_api_endpoints.sh` - Complete API endpoint testing suite with port awareness
  - TESTING_GUIDE.md - Comprehensive testing procedures and troubleshooting (including port conflicts)
- **Documentation**
  - DOCKER_API.md - API deployment guide with examples
  - TESTING_GUIDE.md - Validation procedures and troubleshooting
  - API endpoint documentation with request/response examples
  - Windmill integration patterns and examples

### Changed
- **Architecture** - Transformed from batch-only to API-first service with database persistence
- **Data Storage** - Migrated from JSONL files to SQLite relational database
  - Price data now stored in `price_data` table instead of `merged.jsonl`
  - Tools/price_tools.py updated to query database
  - Position data remains in database (already migrated in earlier versions)
- **Deployment** - Simplified to single API-only Docker service (REST API is new in v0.3.0)
- **Configuration** - Simplified environment variable configuration
  - **Added:** `AUTO_DOWNLOAD_PRICE_DATA` (default: true) - Enable on-demand downloads
  - **Added:** `MAX_SIMULATION_DAYS` (default: 30) - Maximum date range size
  - **Added:** `API_PORT` for host port mapping (default: 8080, customizable for port conflicts)
  - **Removed:** `RUNTIME_ENV_PATH` (API dynamically manages runtime configs)
  - **Removed:** MCP service ports (MATH_HTTP_PORT, SEARCH_HTTP_PORT, TRADE_HTTP_PORT, GETPRICE_HTTP_PORT)
  - **Removed:** `WEB_HTTP_PORT` (web UI not implemented)
  - MCP services use fixed internal ports (8000-8003) and are no longer exposed to host
  - Container always uses port 8080 internally for API
  - Only API port (8080) is exposed to host
  - Reduces configuration complexity and attack surface
- **Requirements** - Added fastapi>=0.120.0, uvicorn[standard]>=0.27.0, pydantic>=2.0.0
- **Docker Compose** - Single service (ai-trader-server) instead of dual-mode
- **Dockerfile** - Added system dependencies (curl, procps) and port 8080 exposure
- **.env.example** - Simplified configuration with only essential variables
- **Entrypoint** - Unified entrypoint.sh with proper signal handling (exec uvicorn)

### Technical Implementation
- **Test-Driven Development** - All components written with tests first
- **Mock-based Testing** - Avoid heavy dependencies in unit tests
- **Pydantic V2** - Type-safe request/response validation
- **Foreign Key Constraints** - Database referential integrity with cascade deletes
- **Thread-safe Execution** - Isolated runtime configs per model-day
- **Background Job Execution** - ThreadPoolExecutor for parallel model execution
- **Automatic Status Transitions** - Job status updates based on model-day completion

### Performance & Quality
- **Test Suite** - 175 tests, all passing
  - Unit tests: 155 tests
  - Integration tests: 18 tests
  - API tests: 20+ tests
- **Code Coverage** - High coverage for new modules
  - Date utilities: 100%
  - Price data manager: 85%
  - Database layer: 98%
  - Job manager: 98%
  - Pydantic models: 100%
  - Runtime manager: 89%
  - Model executor: 84%
  - FastAPI app: 81%
- **Test Execution** - Fast test suite (~12 seconds for full suite)

### Integration Ready
- **Windmill.dev** - HTTP-based integration with polling support
- **External Orchestration** - RESTful API for workflow automation
- **Monitoring** - Health checks and status tracking
- **Persistence** - SQLite database survives container restarts

### Breaking Changes
- **Batch Mode Removed** - All simulations now run through REST API
  - v0.2.0 used sequential batch execution via Docker entrypoint
  - v0.3.0 introduces REST API for external orchestration
  - Migration: Use `POST /simulate/trigger` endpoint instead of direct script execution
- **Data Storage Format Changed** - Price data moved from JSONL to SQLite
  - Run `python scripts/migrate_price_data.py` to migrate existing merged.jsonl data
  - `merged.jsonl` no longer used (replaced by `price_data` table)
  - Automatic on-demand downloads eliminate need for manual data fetching
- **Configuration Variables Changed**
  - Added: `AUTO_DOWNLOAD_PRICE_DATA`, `MAX_SIMULATION_DAYS`, `API_PORT`
  - Removed: `RUNTIME_ENV_PATH`, MCP service ports, `WEB_HTTP_PORT`
  - MCP services now use fixed internal ports (not exposed to host)

## [0.2.0] - 2025-10-31

### Added
- Complete Docker deployment support with containerization
- Docker Compose orchestration for easy local deployment
- Multi-stage Dockerfile with Python 3.10-slim base image
- Automated CI/CD pipeline via GitHub Actions for release builds
- Automatic draft release creation with version tagging
- Docker images published to GitHub Container Registry (ghcr.io)
- Comprehensive Docker documentation (docs/DOCKER.md)
- Release process documentation (docs/RELEASING.md)
- Data cache reuse design documentation (docs/DESIGN_DATA_CACHE_REUSE.md)
- CLAUDE.md repository guidance for development
- Docker deployment section in main README
- Environment variable configuration via docker-compose
- Sequential startup script (entrypoint.sh) for data fetch, MCP services, and trading agent
- Volume mounts for data and logs persistence
- Pre-built image support from ghcr.io/xe138/ai-trader-server
- Configurable volume path for persistent data
- Configurable web interface host port
- Automated merged.jsonl creation during price fetching
- API key registration URLs in .env.example

### Changed
- Updated .env.example with Docker-specific configuration, API key URLs, and paths
- Updated .gitignore to exclude git worktrees directory
- Removed deprecated version tag from docker-compose.yml
- Updated repository URLs to Xe138/AI-Trader-Server fork
- Docker Compose now uses pre-built image by default
- Simplified Docker config file selection with convention over configuration
- Fixed internal ports with configurable host ports
- Separated data scripts from volume mount directory
- Reduced log flooding during data fetch
- OPENAI_API_BASE can now be left empty in configuration

### Fixed
- Docker Compose configuration now follows modern best practices (version-less)
- Prevent restart loop on missing API keys with proper validation
- Docker tag generation now converts repository owner to lowercase
- Validate GITHUB_REF is a tag in docker-release workflow
- Correct Dockerfile FROM AS casing
- Module import errors for MCP services resolved with PYTHONPATH
- Prevent price data overwrite on container restart
- Merge script now writes to current directory for volume compatibility

## [0.1.0] - Initial Release

### Added
- AI trading competition platform for NASDAQ 100 stocks
- Support for multiple AI models (GPT, Claude, Qwen, DeepSeek, Gemini)
- MCP (Model Context Protocol) toolchain integration
  - Mathematical calculation tools
  - Market intelligence search via Jina AI
  - Trading execution tools
  - Price query tools
- Historical replay architecture with anti-look-ahead controls
- Alpha Vantage API integration for price data
- Autonomous AI decision-making with zero human intervention
- Real-time performance analytics and leaderboard
- Position tracking and trading logs
- Web-based performance dashboard
- Complete NASDAQ 100 stock universe support
- Initial capital: $10,000 per AI model
- Configurable date range for backtesting
- Multi-model concurrent trading support
- Automatic data fetching and merging
- Comprehensive README with quick start guide

### Technical Details
- Python 3.10+ support
- LangChain framework integration
- FastMCP for MCP service implementation
- JSONL format for position and log storage
- Weekday-only trading simulation
- Configurable agent parameters (max_steps, max_retries, initial_cash)

---

## Release Notes Template

For future releases, use this template:

```markdown
## [X.Y.Z] - YYYY-MM-DD

### Added
- New features

### Changed
- Changes to existing functionality

### Deprecated
- Soon-to-be removed features

### Removed
- Removed features

### Fixed
- Bug fixes

### Security
- Security improvements
```

---

[Unreleased]: https://github.com/Xe138/AI-Trader-Server/compare/v0.3.0...HEAD
[0.3.0]: https://github.com/Xe138/AI-Trader-Server/compare/v0.2.0...v0.3.0
[0.2.0]: https://github.com/Xe138/AI-Trader-Server/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/Xe138/AI-Trader-Server/releases/tag/v0.1.0
