Compare commits

...

6 Commits

Author SHA1 Message Date
5da02b4ba0 docs: update CHANGELOG.md for v0.2.0 release
Update changelog with comprehensive release notes including:
- All features added during alpha testing phase
- Configuration improvements and new documentation
- Bug fixes and stability improvements
- Corrected release date to 2025-10-31

Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-31 00:28:13 -04:00
11509ba8c7 fix: merge script now writes to current directory for volume compatibility
Changed merge_jsonl.py to use os.getcwd() instead of os.path.dirname(__file__)
to ensure merged.jsonl is written to the working directory where data files exist,
not to the script's installation directory.

Root cause:
- Dockerfile copies scripts to /app/scripts/ for volume compatibility
- entrypoint.sh runs: cd /app/data && python /app/scripts/merge_jsonl.py
- Old logic used script directory (/app/scripts/), ignoring working directory
- This caused merged.jsonl to be created in /app/scripts/ instead of /app/data/
- Since /app/data/ is volume-mounted, merged file was not visible to host

Solution:
- Scripts now respect current working directory (Unix philosophy)
- Works correctly with volume mounts and script relocation
- Tested in both local and Docker directory structure scenarios

Fixes the issue where merged.jsonl was missing from mounted data volume.
2025-10-31 00:03:39 -04:00
1785f9b06f feat: automate merged.jsonl creation during price fetching
Streamline the data preparation workflow by having get_daily_price.py
automatically invoke merge_jsonl.py after fetching all stock prices.

Changes:
- Modified get_daily_price.py to call merge_jsonl.py automatically
- Updated entrypoint.sh to remove redundant merge_jsonl.py call
- Updated main.sh to remove redundant merge_jsonl.py call
- Fixed import order for linting compliance

Benefits:
- Single command now handles both fetching and merging
- Ensures merged.jsonl is always created after price updates
- Simplifies Docker container startup process
- Prevents missing merged.jsonl errors in production
2025-10-30 23:48:12 -04:00
55206549c7 feat: add configurable volume path for persistent data
Add VOLUME_PATH environment variable to customize storage location for
all persistent data (data/, logs/, configs/ subdirectories).

Changes:
- .env.example: Add VOLUME_PATH variable with documentation
- docker-compose.yml: Use ${VOLUME_PATH:-.} for all volume mounts
- docs/DOCKER.md: Document custom volume location feature

Benefits:
- Store data outside project directory (e.g., on separate disk)
- Easier backup/restore (single directory)
- Clean separation of code and data
- Support for absolute or relative paths

Usage:
  # In .env file
  VOLUME_PATH=/home/user/trading-data

  # Results in mounts:
  /home/user/trading-data/data:/app/data
  /home/user/trading-data/logs:/app/logs
  /home/user/trading-data/configs:/app/configs

Defaults to current directory (.) if not set for backward compatibility.
2025-10-30 23:40:21 -04:00
9e05ce0891 fix: prevent price data overwrite on container restart
Preserve existing merged.jsonl to avoid data loss and API rate limits.
Only fetch new data if merged.jsonl is missing or empty.

Problem:
- Entrypoint always fetched fresh data from Alpha Vantage on every start
- Overwrote existing mounted data directory
- Caused API rate limit issues and data inconsistencies
- Lost historical data needed for backtesting specific date ranges

Solution:
- Check if merged.jsonl exists and has content before fetching
- Display stock count when using existing data
- Provide manual refresh instructions for when updates are needed

Benefits:
- Faster container startup (no API calls if data exists)
- Avoids Alpha Vantage rate limits (5 calls/min, 500/day)
- Preserves user's existing historical datasets
- Enables reliable backtesting with consistent data

To refresh data: rm data/merged.jsonl && docker-compose restart
2025-10-30 23:27:58 -04:00
2f2c1d6ea2 feat: simplify Docker config file selection with convention over configuration
Implement automatic detection of custom_config.json for simpler Docker usage.
No environment variables or command-line arguments needed for most users.

Changes:
- entrypoint.sh: Add smart config detection (custom_config.json > CLI arg > default_config.json)
- docker-compose.yml: Add configs volume mount for editing without rebuilds
- docs/DOCKER.md: Update documentation with simplified workflow
- .gitignore: Add custom_config.json to prevent accidental commits

Usage:
  cp configs/default_config.json configs/custom_config.json
  nano configs/custom_config.json
  docker-compose up  # Automatically uses custom_config.json

Simplifies user experience by following convention over configuration principle.
2025-10-30 22:34:22 -04:00
9 changed files with 171 additions and 30 deletions

View File

@@ -30,3 +30,9 @@ WEB_HTTP_PORT=8888
# Agent Configuration
AGENT_MAX_STEP=30
# Data Volume Configuration
# Base directory for all persistent data (will contain data/, logs/, configs/ subdirectories)
# Use relative paths (./volumes) or absolute paths (/home/user/ai-trader-volumes)
# Defaults to current directory (.) if not set
VOLUME_PATH=.

1
.gitignore vendored
View File

@@ -57,6 +57,7 @@ delete.py
refresh_data.sh
# Config files (optional - uncomment if needed)
configs/custom_config.json
configs/day_config.json
configs/hour_config.json
configs/test_config.json

View File

@@ -7,30 +7,50 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.2.0] - 2025-10-30
## [0.2.0] - 2025-10-31
### Added
- Complete Docker deployment support with containerization
- Docker Compose orchestration for easy local deployment
- Multi-stage Dockerfile with Python 3.10-slim base image
- Automated CI/CD pipeline via GitHub Actions for release builds
- Automatic draft release creation with version tagging
- Docker images published to GitHub Container Registry (ghcr.io)
- Comprehensive Docker documentation (docs/DOCKER.md)
- Release process documentation (docs/RELEASING.md)
- Data cache reuse design documentation (docs/DESIGN_DATA_CACHE_REUSE.md)
- CLAUDE.md repository guidance for development
- Docker deployment section in main README
- Environment variable configuration via docker-compose
- Sequential startup script (entrypoint.sh) for data fetch, MCP services, and trading agent
- Volume mounts for data and logs persistence
- Pre-built image support from ghcr.io/hkuds/ai-trader
- Pre-built image support from ghcr.io/xe138/ai-trader
- Configurable volume path for persistent data
- Configurable web interface host port
- Automated merged.jsonl creation during price fetching
- API key registration URLs in .env.example
### Changed
- Updated .env.example with Docker-specific configuration and paths
- Updated .env.example with Docker-specific configuration, API key URLs, and paths
- Updated .gitignore to exclude git worktrees directory
- Removed deprecated version tag from docker-compose.yml
- Updated repository URLs to Xe138/AI-Trader fork
- Docker Compose now uses pre-built image by default
- Simplified Docker config file selection with convention over configuration
- Fixed internal ports with configurable host ports
- Separated data scripts from volume mount directory
- Reduced log flooding during data fetch
- OPENAI_API_BASE can now be left empty in configuration
### Fixed
- Docker Compose configuration now follows modern best practices (version-less)
- Prevent restart loop on missing API keys with proper validation
- Docker tag generation now converts repository owner to lowercase
- Validate GITHUB_REF is a tag in docker-release workflow
- Correct Dockerfile FROM AS casing
- Module import errors for MCP services resolved with PYTHONPATH
- Prevent price data overwrite on container restart
- Merge script now writes to current directory for volume compatibility
## [0.1.0] - Initial Release

View File

@@ -1,8 +1,12 @@
import requests
import os
from dotenv import load_dotenv
load_dotenv()
import json
import os
import subprocess
import sys
import requests
from dotenv import load_dotenv
load_dotenv()
all_nasdaq_100_symbols = [
@@ -42,4 +46,15 @@ if __name__ == "__main__":
for symbol in all_nasdaq_100_symbols:
get_daily_price(symbol)
get_daily_price("QQQ")
get_daily_price("QQQ")
# Automatically run merge after fetching
print("\n📦 Merging price data...")
try:
script_dir = os.path.dirname(os.path.abspath(__file__))
merge_script = os.path.join(script_dir, "merge_jsonl.py")
subprocess.run([sys.executable, merge_script], check=True)
print("✅ Price data merged successfully")
except Exception as e:
print(f"⚠️ Failed to merge data: {e}")
print(" Please run 'python merge_jsonl.py' manually")

View File

@@ -18,7 +18,8 @@ all_nasdaq_100_symbols = [
]
# 合并所有以 daily_price 开头的 json逐文件一行写入 merged.jsonl
current_dir = os.path.dirname(__file__)
# Use current working directory instead of script directory for volume compatibility
current_dir = os.getcwd()
pattern = os.path.join(current_dir, 'daily_price*.json')
files = sorted(glob.glob(pattern))

View File

@@ -5,8 +5,9 @@ services:
# build: .
container_name: ai-trader-app
volumes:
- ./data:/app/data
- ./logs:/app/logs
- ${VOLUME_PATH:-.}/data:/app/data
- ${VOLUME_PATH:-.}/logs:/app/logs
- ${VOLUME_PATH:-.}/configs:/app/configs
environment:
# AI Model API Configuration
- OPENAI_API_BASE=${OPENAI_API_BASE}

View File

@@ -53,10 +53,30 @@ AGENT_MAX_STEP=30
### Custom Trading Configuration
Pass a custom config file:
**Simple Method (Recommended):**
Create a `configs/custom_config.json` file - it will be automatically used:
```bash
docker-compose run ai-trader configs/my_config.json
# Copy default config as starting point
cp configs/default_config.json configs/custom_config.json
# Edit your custom config
nano configs/custom_config.json
# Run normally - custom_config.json is automatically detected!
docker-compose up
```
**Priority order:**
1. `configs/custom_config.json` (if exists) - **Highest priority**
2. Command-line argument: `docker-compose run ai-trader configs/other.json`
3. `configs/default_config.json` (fallback)
**Advanced: Use a different config file name:**
```bash
docker-compose run ai-trader configs/my_special_config.json
```
## Usage Examples
@@ -92,16 +112,43 @@ docker-compose up
### Volume Mounts
Docker Compose mounts two volumes:
Docker Compose mounts three volumes for persistent data. By default, these are stored in the project directory:
- `./data:/app/data` - Price data and trading records
- `./logs:/app/logs` - MCP service logs
- `./configs:/app/configs` - Configuration files (allows editing configs without rebuilding)
Data persists across container restarts. To reset:
### Custom Volume Location
You can change where data is stored by setting `VOLUME_PATH` in your `.env` file:
```bash
# Store data in a different location
VOLUME_PATH=/home/user/trading-data
# Or use a relative path
VOLUME_PATH=./volumes
```
This will store data in:
- `/home/user/trading-data/data/`
- `/home/user/trading-data/logs/`
- `/home/user/trading-data/configs/`
**Note:** The directory structure is automatically created. You'll need to copy your existing configs:
```bash
# After changing VOLUME_PATH
mkdir -p /home/user/trading-data/configs
cp configs/custom_config.json /home/user/trading-data/configs/
```
### Reset Data
To reset all trading data:
```bash
docker-compose down
rm -rf data/agent_data/* logs/*
rm -rf ${VOLUME_PATH:-.}/data/agent_data/* ${VOLUME_PATH:-.}/logs/*
docker-compose up
```
@@ -227,13 +274,45 @@ Services exposed on host:
### Test Different Configurations
```bash
# Create test config
cp configs/default_config.json configs/test_config.json
# Edit test_config.json
**Method 1: Use the standard custom_config.json**
# Run with test config
docker-compose run ai-trader configs/test_config.json
```bash
# Create and edit your config
cp configs/default_config.json configs/custom_config.json
nano configs/custom_config.json
# Run - automatically uses custom_config.json
docker-compose up
```
**Method 2: Test multiple configs with different names**
```bash
# Create multiple test configs
cp configs/default_config.json configs/conservative.json
cp configs/default_config.json configs/aggressive.json
# Edit each config...
# Test conservative strategy
docker-compose run ai-trader configs/conservative.json
# Test aggressive strategy
docker-compose run ai-trader configs/aggressive.json
```
**Method 3: Temporarily switch configs**
```bash
# Temporarily rename your custom config
mv configs/custom_config.json configs/custom_config.json.backup
cp configs/test_strategy.json configs/custom_config.json
# Run with test strategy
docker-compose up
# Restore original
mv configs/custom_config.json.backup configs/custom_config.json
```
## Production Deployment

View File

@@ -38,12 +38,18 @@ fi
echo "✅ Environment variables validated"
# Step 1: Data preparation
echo "📊 Fetching and merging price data..."
# Run scripts from /app/scripts but output to /app/data
cd /app/data
python /app/scripts/get_daily_price.py
python /app/scripts/merge_jsonl.py
cd /app
echo "📊 Checking price data..."
if [ -f "/app/data/merged.jsonl" ] && [ -s "/app/data/merged.jsonl" ]; then
echo "✅ Using existing price data ($(wc -l < /app/data/merged.jsonl) stocks)"
echo " To refresh data, delete /app/data/merged.jsonl and restart"
else
echo "📊 Fetching and merging price data..."
# Run script from /app/scripts but output to /app/data
# Note: get_daily_price.py now automatically calls merge_jsonl.py after fetching
cd /app/data
python /app/scripts/get_daily_price.py
cd /app
fi
# Step 2: Start MCP services in background
echo "🔧 Starting MCP services..."
@@ -57,7 +63,19 @@ sleep 3
# Step 4: Run trading agent with config file
echo "🤖 Starting trading agent..."
CONFIG_FILE="${1:-configs/default_config.json}"
# Smart config selection: custom_config.json takes precedence if it exists
if [ -f "configs/custom_config.json" ]; then
CONFIG_FILE="configs/custom_config.json"
echo "✅ Using custom configuration: configs/custom_config.json"
elif [ -n "$1" ]; then
CONFIG_FILE="$1"
echo "✅ Using specified configuration: $CONFIG_FILE"
else
CONFIG_FILE="configs/default_config.json"
echo "✅ Using default configuration: configs/default_config.json"
fi
python main.py "$CONFIG_FILE"
# Cleanup on exit

View File

@@ -10,8 +10,8 @@ echo "🚀 Launching AI Trader Environment..."
echo "📊 Now getting and merging price data..."
cd ./data
# Note: get_daily_price.py now automatically calls merge_jsonl.py after fetching
python get_daily_price.py
python merge_jsonl.py
cd ../
echo "🔧 Now starting MCP services..."