Commit Graph

132 Commits

Author SHA1 Message Date
47b9df6b82 docs: merge unreleased configuration changes into v0.3.0
Consolidated the configuration simplification changes (RUNTIME_ENV_PATH
removal, API_PORT cleanup, MCP port removal) into the v0.3.0 release
notes under the 'Changed - Configuration' section.

This ensures all v0.3.0 changes are documented together in a single
release entry rather than split across Unreleased and v0.3.0 sections.
v0.3.0-alpha.2
2025-10-31 14:43:31 -04:00
d587a5f213 refactor: remove unnecessary MCP service port configuration
MCP services are completely internal to the container and accessed
only via localhost. They should not be configurable or exposed.

Changes:
- Remove MATH_HTTP_PORT, SEARCH_HTTP_PORT, TRADE_HTTP_PORT,
  GETPRICE_HTTP_PORT from docker-compose.yml environment
- Remove MCP service port mappings from docker-compose.yml
- Remove MCP port configuration from .env.example
- Update README.md to remove MCP port configuration
- Update CLAUDE.md to clarify MCP services use fixed internal ports
- Update CHANGELOG.md with these simplifications

Technical details:
- MCP services hardcode to ports 8000-8003 via os.getenv() defaults
- Services only accessed via localhost URLs within container:
  - http://localhost:8000/mcp (math)
  - http://localhost:8001/mcp (search)
  - http://localhost:8002/mcp (trade)
  - http://localhost:8003/mcp (price)
- No external access needed or desired for these services
- Only API (8080) and web dashboard (8888) should be exposed

Benefits:
- Simpler configuration (4 fewer environment variables)
- Reduced attack surface (4 fewer exposed ports)
- Clearer architecture (internal vs external services)
- Prevents accidental misconfiguration of internal services
2025-10-31 14:41:07 -04:00
c929080960 fix: remove API_PORT from container environment variables
The API_PORT variable was incorrectly included in the container's
environment section. It should only be used for host port mapping
in docker-compose.yml, not passed into the container.

Changes:
- Remove API_PORT from environment section in docker-compose.yml
- Container always uses port 8080 internally (hardcoded in entrypoint.sh)
- API_PORT in .env/.env.example only controls the host-side mapping:
  ports: "${API_PORT:-8080}:8080" (host:container)

Why this matters:
- Prevents confusion about whether API_PORT changes internal port
- Clarifies that entrypoint.sh hardcodes --port 8080
- Simplifies container environment (one less unused variable)
- More explicit about the port mapping behavior

No functional change - the container was already ignoring this variable.
2025-10-31 14:38:53 -04:00
849e7bffa2 refactor: remove unnecessary RUNTIME_ENV_PATH environment variable
Simplifies deployment configuration by removing the RUNTIME_ENV_PATH
environment variable, which is no longer needed for API mode.

Changes:
- Remove RUNTIME_ENV_PATH from docker-compose.yml
- Remove RUNTIME_ENV_PATH from .env.example
- Update CLAUDE.md to reflect API-managed runtime configs
- Update README.md to remove RUNTIME_ENV_PATH from config examples
- Update CHANGELOG.md with this simplification

Technical details:
- API mode dynamically creates isolated runtime config files via
  RuntimeConfigManager (data/runtime_env_{job_id}_{model}_{date}.json)
- tools/general_tools.py already handles missing RUNTIME_ENV_PATH
  gracefully, returning empty dict and warning on writes
- No functional impact - all tests pass without this variable set
- Reduces configuration complexity for new deployments

Breaking change: None - variable was vestigial from batch mode era
2025-10-31 14:37:00 -04:00
c17b3db29d feat: implement conditional Docker tagging for pre-releases
Updates the docker-release.yml workflow to distinguish between
stable releases and pre-releases (alpha, beta, rc versions).

Changes:
- Add pre-release detection logic to extract version step
- Create new "Generate Docker tags" step to conditionally build tag list
- Only apply 'latest' tag for stable releases
- Pre-releases are tagged with version number only
- Update "Image published" message to reflect pre-release status

Example behavior:
- v0.3.0 -> tags: 0.3.0, latest
- v0.3.0-alpha -> tags: 0.3.0-alpha (NOT latest)
- v1.0.0-rc1 -> tags: 1.0.0-rc1 (NOT latest)

This prevents pre-release versions from overwriting the stable
'latest' tag, allowing users to safely pull the latest stable
version while still providing access to pre-release versions by
explicit version tag.
v0.3.0-alpha
2025-10-31 14:28:43 -04:00
cf6b56247e docs: merge unreleased changes into v0.3.0 release notes
- Consolidated batch mode removal into v0.3.0
- Updated deployment description to API-only
- Added breaking changes section
- Documented port configuration enhancements
- Added system dependencies (curl, procps)
- Removed outdated dual-mode references
- Ready for v0.3.0 release
2025-10-31 14:21:56 -04:00
483eca7c77 docs: add port configuration troubleshooting
- Document port conflict resolution in TESTING_GUIDE.md
- Add example for custom API_PORT in .env.example
- Explain container vs host port architecture
- Provide solutions for common port conflict scenarios
2025-10-31 14:18:48 -04:00
b88a65d9d7 fix: API endpoint test script now reads API_PORT from .env
- Read API_PORT from .env file if it exists
- Construct API_BASE_URL using configured port
- Display which URL is being tested
- Consistent with validate_docker_build.sh behavior
2025-10-31 14:15:48 -04:00
71829918ca fix: validation script now reads API_PORT from .env
- Read API_PORT from .env file if it exists
- Use configured port instead of hardcoded 8080
- Display which port is being tested
- Fixes validation when API_PORT is customized
2025-10-31 14:13:53 -04:00
2623bdaca4 fix: install curl and procps in Docker image for health checks
- Add curl for Docker health checks and diagnostics
- Add procps for process monitoring (ps command)
- Required for validation scripts to work properly
- Minimal size increase (~5MB) for critical debugging tools
2025-10-31 14:06:00 -04:00
68867e407e debug: add FastAPI app import check before starting uvicorn 2025-10-31 14:03:30 -04:00
ceb2eabff9 fix: correct entrypoint script trap and uvicorn execution
- Move trap setup before uvicorn (was after, never executed)
- Use exec to replace bash with uvicorn process (better signal handling)
- Ensures uvicorn stays running as PID 1 in container
2025-10-31 14:00:57 -04:00
cfa2428393 fix: improve health check validation with retries and diagnostics
- Add retry logic (up to 15 attempts over 30 seconds)
- Add comprehensive diagnostics on failure
- Test endpoint from inside container to isolate networking issues
- Show recent logs if health check fails
- Better error messages for troubleshooting
2025-10-31 13:58:20 -04:00
357e561b1f refactor: remove batch mode, simplify to API-only deployment
Removes dual-mode deployment complexity, focusing on REST API service only.

Changes:
- Removed batch mode from docker-compose.yml (now single ai-trader service)
- Deleted scripts/test_batch_mode.sh validation script
- Renamed entrypoint-api.sh to entrypoint.sh (now default)
- Simplified Dockerfile (single entrypoint, removed CMD)
- Updated validation scripts to use 'ai-trader' service name
- Updated documentation (README.md, TESTING_GUIDE.md, CHANGELOG.md)

Benefits:
- Eliminates port conflicts between batch and API services
- Simpler configuration and deployment
- API-first architecture aligned with Windmill integration
- Reduced maintenance complexity

Breaking Changes:
- Batch mode no longer available
- All simulations must use REST API endpoints
2025-10-31 13:54:14 -04:00
a9f9560f76 fix: add jobs.db to .gitignore 2025-10-31 12:45:40 -04:00
eac2e781f7 docs: clarify API_PORT usage in .env.example
Added detailed comments explaining that container always uses port 8080
internally and API_PORT only controls host port mapping.
2025-10-31 12:42:41 -04:00
77da47a40d fix: use hardcoded port 8080 internally in container
API_PORT env var now only controls host port mapping in docker-compose.yml.
Container always binds uvicorn to port 8080 internally for consistency
with health checks and documentation.
2025-10-31 12:42:28 -04:00
c63cdffd0e fix: enable local Docker builds for development and testing 2025-10-31 12:37:41 -04:00
1c8e59340e fix: make entrypoint-api.sh executable for Docker 2025-10-31 12:34:51 -04:00
fb9583b374 feat: transform to REST API service with SQLite persistence (v0.3.0)
Major architecture transformation from batch-only to API service with
database persistence for Windmill integration.

## REST API Implementation
- POST /simulate/trigger - Start simulation jobs
- GET /simulate/status/{job_id} - Monitor job progress
- GET /results - Query results with filters (job_id, date, model)
- GET /health - Service health checks

## Database Layer
- SQLite persistence with 6 tables (jobs, job_details, positions,
  holdings, reasoning_logs, tool_usage)
- Foreign key constraints with cascade deletes
- Replaces JSONL file storage

## Backend Components
- JobManager: Job lifecycle management with concurrency control
- RuntimeConfigManager: Thread-safe isolated runtime configs
- ModelDayExecutor: Single model-day execution engine
- SimulationWorker: Date-sequential, model-parallel orchestration

## Testing
- 102 unit and integration tests (85% coverage)
- Database: 98% coverage
- Job manager: 98% coverage
- API endpoints: 81% coverage
- Pydantic models: 100% coverage
- TDD approach throughout

## Docker Deployment
- Dual-mode: API server (persistent) + batch (one-time)
- Health checks with 30s interval
- Volume persistence for database and logs
- Separate entrypoints for each mode

## Validation Tools
- scripts/validate_docker_build.sh - Build validation
- scripts/test_api_endpoints.sh - Complete API testing
- scripts/test_batch_mode.sh - Batch mode validation
- DOCKER_API.md - Deployment guide
- TESTING_GUIDE.md - Testing procedures

## Configuration
- API_PORT environment variable (default: 8080)
- Backwards compatible with existing configs
- FastAPI, uvicorn, pydantic>=2.0 dependencies

Co-Authored-By: AI Assistant <noreply@example.com>
2025-10-31 11:47:10 -04:00
5da02b4ba0 docs: update CHANGELOG.md for v0.2.0 release
Update changelog with comprehensive release notes including:
- All features added during alpha testing phase
- Configuration improvements and new documentation
- Bug fixes and stability improvements
- Corrected release date to 2025-10-31

Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
v0.2.0
2025-10-31 00:28:13 -04:00
11509ba8c7 fix: merge script now writes to current directory for volume compatibility
Changed merge_jsonl.py to use os.getcwd() instead of os.path.dirname(__file__)
to ensure merged.jsonl is written to the working directory where data files exist,
not to the script's installation directory.

Root cause:
- Dockerfile copies scripts to /app/scripts/ for volume compatibility
- entrypoint.sh runs: cd /app/data && python /app/scripts/merge_jsonl.py
- Old logic used script directory (/app/scripts/), ignoring working directory
- This caused merged.jsonl to be created in /app/scripts/ instead of /app/data/
- Since /app/data/ is volume-mounted, merged file was not visible to host

Solution:
- Scripts now respect current working directory (Unix philosophy)
- Works correctly with volume mounts and script relocation
- Tested in both local and Docker directory structure scenarios

Fixes the issue where merged.jsonl was missing from mounted data volume.
v0.2.0-alpha.11
2025-10-31 00:03:39 -04:00
1785f9b06f feat: automate merged.jsonl creation during price fetching
Streamline the data preparation workflow by having get_daily_price.py
automatically invoke merge_jsonl.py after fetching all stock prices.

Changes:
- Modified get_daily_price.py to call merge_jsonl.py automatically
- Updated entrypoint.sh to remove redundant merge_jsonl.py call
- Updated main.sh to remove redundant merge_jsonl.py call
- Fixed import order for linting compliance

Benefits:
- Single command now handles both fetching and merging
- Ensures merged.jsonl is always created after price updates
- Simplifies Docker container startup process
- Prevents missing merged.jsonl errors in production
v0.2.0-alpha.10
2025-10-30 23:48:12 -04:00
55206549c7 feat: add configurable volume path for persistent data
Add VOLUME_PATH environment variable to customize storage location for
all persistent data (data/, logs/, configs/ subdirectories).

Changes:
- .env.example: Add VOLUME_PATH variable with documentation
- docker-compose.yml: Use ${VOLUME_PATH:-.} for all volume mounts
- docs/DOCKER.md: Document custom volume location feature

Benefits:
- Store data outside project directory (e.g., on separate disk)
- Easier backup/restore (single directory)
- Clean separation of code and data
- Support for absolute or relative paths

Usage:
  # In .env file
  VOLUME_PATH=/home/user/trading-data

  # Results in mounts:
  /home/user/trading-data/data:/app/data
  /home/user/trading-data/logs:/app/logs
  /home/user/trading-data/configs:/app/configs

Defaults to current directory (.) if not set for backward compatibility.
v0.2.0-alpha.9
2025-10-30 23:40:21 -04:00
9e05ce0891 fix: prevent price data overwrite on container restart
Preserve existing merged.jsonl to avoid data loss and API rate limits.
Only fetch new data if merged.jsonl is missing or empty.

Problem:
- Entrypoint always fetched fresh data from Alpha Vantage on every start
- Overwrote existing mounted data directory
- Caused API rate limit issues and data inconsistencies
- Lost historical data needed for backtesting specific date ranges

Solution:
- Check if merged.jsonl exists and has content before fetching
- Display stock count when using existing data
- Provide manual refresh instructions for when updates are needed

Benefits:
- Faster container startup (no API calls if data exists)
- Avoids Alpha Vantage rate limits (5 calls/min, 500/day)
- Preserves user's existing historical datasets
- Enables reliable backtesting with consistent data

To refresh data: rm data/merged.jsonl && docker-compose restart
v0.2.0-alpha.8
2025-10-30 23:27:58 -04:00
2f2c1d6ea2 feat: simplify Docker config file selection with convention over configuration
Implement automatic detection of custom_config.json for simpler Docker usage.
No environment variables or command-line arguments needed for most users.

Changes:
- entrypoint.sh: Add smart config detection (custom_config.json > CLI arg > default_config.json)
- docker-compose.yml: Add configs volume mount for editing without rebuilds
- docs/DOCKER.md: Update documentation with simplified workflow
- .gitignore: Add custom_config.json to prevent accidental commits

Usage:
  cp configs/default_config.json configs/custom_config.json
  nano configs/custom_config.json
  docker-compose up  # Automatically uses custom_config.json

Simplifies user experience by following convention over configuration principle.
v0.2.0-alpha.7
2025-10-30 22:34:22 -04:00
e9c571402a fix: set PYTHONPATH for MCP services to resolve import errors
Root cause: Python adds script directory (not working directory)
to sys.path. Services in /app/agent_tools/ couldn't import from
/app/tools/ because sys.path[0] was /app/agent_tools.

Solution: Set PYTHONPATH=/app when starting services so /app is
always in the Python import path.

Fixes: ModuleNotFoundError: No module named 'tools' in price.log
v0.2.0-alpha.6
2025-10-30 21:47:58 -04:00
79d14444ed docs: add data cache reuse design document
Captures design for staleness-based data refresh to avoid
re-fetching all 103 NASDAQ tickers on every container restart.

Key features:
- Check all daily_prices_*.json files for staleness
- Configurable MAX_DATA_AGE_DAYS threshold (default: 7)
- Bash wrapper logic in entrypoint.sh
- Handles edge cases (partial data, missing files, rate limits)
2025-10-30 21:46:05 -04:00
12ecb1e6b6 docs: clarify OPENAI_API_BASE can be left empty
Add comment explaining OPENAI_API_BASE can be left empty
to use the default OpenAI endpoint, or set to a custom
proxy URL if needed.

Sets default value to empty instead of placeholder URL.
v0.2.0-alpha.5
2025-10-30 21:34:16 -04:00
203b60b252 Revert "fix: improve MCP service startup reliability"
This reverts commit d70362b9d4.
2025-10-30 21:33:12 -04:00
d70362b9d4 fix: improve MCP service startup reliability
- Clarify OPENAI_API_BASE can be left empty for default endpoint
- Increase initial wait time from 3s to 5s
- Add health checking for all MCP services (ports 8000-8003)
- Retry up to 10 times with 1s intervals
- Install netcat for port checking
- Display warnings if services aren't ready

Helps diagnose and prevent MCP client initialization errors.
2025-10-30 21:32:42 -04:00
5ec7977b47 fix: resolve module import error for MCP services
Run MCP service manager from /app instead of /app/agent_tools
to ensure services can import from tools module.

Changes:
- entrypoint.sh: Run start_mcp_services.py from /app directory
- start_mcp_services.py: Use absolute paths for service scripts
- start_mcp_services.py: Set working directory to /app for services

Fixes ModuleNotFoundError: No module named 'tools' in price.log
2025-10-30 21:22:42 -04:00
8142f38ab9 docs: add API key registration URLs to .env.example
Add inline comments with URLs for API key registration:
- OPENAI_API_KEY: https://platform.openai.com/api-keys
- ALPHAADVANTAGE_API_KEY: https://www.alphavantage.co/support/#api-key
- JINA_API_KEY: https://jina.ai/

Makes it easier for users to find where to get their API keys.
v0.2.0-alpha.4
2025-10-30 21:17:21 -04:00
595a659fe7 fix: reduce log flooding during data fetch
Replace verbose JSON logging with concise status messages:
- Success: '✓ Fetched SYMBOL'
- Error: '⚠️  SYMBOL: API rate limit or error - [message]'

Prevents logs from being flooded with full JSON responses
for 100+ stock symbols during startup.
2025-10-30 21:15:59 -04:00
a4bc4fd0de fix: update repository URLs to Xe138/AI-Trader fork
Update all GitHub URLs in README from HKUDS/AI-Trader to
Xe138/AI-Trader including:
- Clone URLs (2 locations)
- Support & Community links (Discussions, Issues)
- Badge URLs (stars, forks)
2025-10-30 21:06:28 -04:00
4666e09385 fix: prevent restart loop on missing API keys
Add validation at startup to check required environment variables:
- OPENAI_API_KEY
- ALPHAADVANTAGE_API_KEY
- JINA_API_KEY

If any are missing, display clear error message with setup
instructions and exit immediately (no restart loop).

Change restart policy from 'unless-stopped' to 'on-failure:3'
to limit restart attempts and prevent endless loops on
configuration errors.
v0.2.0-alpha.3
2025-10-30 21:00:12 -04:00
0c58baed42 fix: separate data scripts from volume mount directory
Move get_daily_price.py and merge_jsonl.py to /app/scripts
to prevent volume mount from overlaying the scripts.
Scripts run from /app/data context to output correctly.

Fixes container startup error where scripts were not found
after ./data volume mount replaced container's /app/data.
v0.2.0-alpha.2
2025-10-30 20:53:44 -04:00
2d1c356199 feat: add configurable web interface host port
Add WEB_HTTP_PORT environment variable to configure
the host port for the web interface (port 8888).
Container still uses 8888 internally, but host port
can be customized via .env file.
2025-10-30 20:51:21 -04:00
ffa158224d fix: use fixed internal ports with configurable host ports
Internal container ports are now fixed at 8000-8003.
Host ports are configurable via .env variables.
This prevents port conflicts inside the container while
allowing users to map to different host ports if needed.
2025-10-30 20:50:37 -04:00
6825a60b20 feat: use pre-built image by default in docker-compose
Pull ghcr.io/xe138/ai-trader:latest by default instead of building.
Add commented build option for local development.
Makes deployment faster for end users.
2025-10-30 20:47:11 -04:00
9e308ee243 fix: use generic template for draft releases
Replace hardcoded release notes with generic template.
User can complete the release notes before publishing.
Includes TODO markers for summary and changelog content.
v0.2.0-alpha
2025-10-30 20:40:30 -04:00
90ea10c7ef feat: add automatic draft release creation to workflow
Automatically create GitHub draft release with comprehensive
release notes when Docker build succeeds. Includes:
- Quick start instructions for Docker Compose and pre-built images
- Full changelog for v0.2.0
- Documentation links
- Automatic pre-release detection for alpha/beta/rc tags
2025-10-30 20:38:59 -04:00
77ce1b2b11 fix: correct Dockerfile FROM AS casing
Use uppercase AS keyword to match FROM keyword casing,
following Docker best practices and eliminating build warning.
2025-10-30 20:36:40 -04:00
2585045be1 fix: convert repository owner to lowercase for Docker tags
Docker tags require lowercase repository names. Convert
github.repository_owner to lowercase before using in tags.
2025-10-30 20:34:43 -04:00
9d5f449b1c fix: validate GITHUB_REF is a tag in docker-release workflow
Add validation to ensure workflow only processes tag pushes.
Prevents invalid Docker tags when workflow runs on non-tag refs.
2025-10-30 20:31:55 -04:00
e4b7e197d3 Update repository URLs for fork
Change HKUDS/AI-Trader to Xe138/AI-Trader
Update all documentation links and references
2025-10-30 20:26:18 -04:00
928f5fb53f Update CHANGELOG for v0.2.0 release
Set version to 0.2.0 for Docker deployment feature
Release date: 2025-10-30
Update comparison links for proper version tracking
2025-10-30 20:24:46 -04:00
46582d38bb Add CHANGELOG.md
Document all changes including Docker deployment feature
Follow Keep a Changelog format with semantic versioning
Include template for future releases
2025-10-30 20:22:10 -04:00
147fab11d5 Remove deprecated version tag from docker-compose.yml
Docker Compose no longer requires version specification
Maintains compatibility with modern docker-compose versions
2025-10-30 20:20:49 -04:00
85fc925b4e Add release process documentation
Complete guide for creating releases:
- Tag creation and push process
- GitHub Actions automation workflow
- Verification and testing steps
- Troubleshooting common issues
2025-10-30 19:26:56 -04:00