diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..dd3f896 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,284 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +AI-Trader is an autonomous AI trading competition platform where multiple AI models compete in NASDAQ 100 trading with zero human intervention. Each AI starts with $10,000 and uses standardized MCP (Model Context Protocol) tools to make fully autonomous trading decisions. + +**Key Innovation:** Historical replay architecture with anti-look-ahead controls ensures AI agents can only access data from the current simulation date and earlier. + +## Development Commands + +### Environment Setup +```bash +# Install dependencies +pip install -r requirements.txt + +# Configure environment variables +cp .env.example .env +# Edit .env and set: +# - OPENAI_API_BASE, OPENAI_API_KEY +# - ALPHAADVANTAGE_API_KEY, JINA_API_KEY +# - RUNTIME_ENV_PATH (recommended: absolute path to runtime_env.json) +# - MCP service ports (default: 8000-8003) +# - AGENT_MAX_STEP (default: 30) +``` + +### Data Preparation +```bash +# Download/update NASDAQ 100 stock data +cd data +python get_daily_price.py # Fetch daily prices from Alpha Vantage +python merge_jsonl.py # Merge into unified format (merged.jsonl) +cd .. +``` + +### Starting Services +```bash +# Start all MCP services (Math, Search, Trade, LocalPrices) +cd agent_tools +python start_mcp_services.py +cd .. + +# Services run on ports defined in .env: +# - MATH_HTTP_PORT (default: 8000) +# - SEARCH_HTTP_PORT (default: 8001) +# - TRADE_HTTP_PORT (default: 8002) +# - GETPRICE_HTTP_PORT (default: 8003) +``` + +### Running Trading Simulations +```bash +# Run with default config +python main.py + +# Run with custom config +python main.py configs/my_config.json + +# Environment variables can override config dates: +INIT_DATE=2025-01-01 END_DATE=2025-01-31 python main.py +``` + +### Complete Workflow +```bash +# All-in-one startup script (data + services + trading + web) +bash main.sh +``` + +## Architecture + +### Core Components + +**1. Agent System** (`agent/base_agent/base_agent.py`) +- `BaseAgent`: Base class for all trading agents +- Manages MCP tool connections, AI model initialization, trading execution loops +- Handles position management and logging +- Supports retry logic with exponential backoff (`max_retries`, `base_delay`) + +**2. Main Entry Point** (`main.py`) +- Dynamic agent class loading via `AGENT_REGISTRY` +- Multi-model concurrent trading support +- Date range validation and weekday filtering +- Configuration management (JSON + environment variables) + +**3. MCP Toolchain** (`agent_tools/`) +- `tool_math.py`: Mathematical calculations (port 8000) +- `tool_jina_search.py`: Market intelligence search (port 8001) +- `tool_trade.py`: Buy/sell execution (port 8002) +- `tool_get_price_local.py`: Price queries (port 8003) +- `start_mcp_services.py`: Service orchestration with health checks + +**4. Data Management** (`data/`) +- `daily_prices_*.json`: Individual stock OHLCV data +- `merged.jsonl`: Unified price data format +- `agent_data/[signature]/position/position.jsonl`: Position records +- `agent_data/[signature]/log/[date]/log.jsonl`: Trading logs + +**5. Utilities** (`tools/`) +- `general_tools.py`: Config management, message extraction +- `price_tools.py`: Price queries, position updates +- `result_tools.py`: Performance calculations + +### Data Flow + +1. **Initialization**: Agent loads config, connects to MCP services, initializes AI model +2. **Trading Loop**: For each date: + - Get system prompt with current positions, yesterday's prices, today's buy prices + - AI agent analyzes market, calls search/math/price tools + - Makes buy/sell decisions via trade tool + - Logs all decisions and updates position.jsonl +3. **Position Tracking**: Each trade appends to `position.jsonl` with date, action, and updated holdings + +### Configuration System + +**Multi-layered config priority:** +1. Environment variables (highest) +2. Model-specific config (`openai_base_url`, `openai_api_key` in model config) +3. JSON config file +4. Default values (lowest) + +**Runtime configuration** (`runtime_env.json` at `RUNTIME_ENV_PATH`): +- Dynamic state: `TODAY_DATE`, `SIGNATURE`, `IF_TRADE` +- Written by `write_config_value()`, read by `get_config_value()` + +### Agent System + +**BaseAgent Key Methods:** +- `initialize()`: Connect to MCP services, create AI model +- `run_trading_session(date)`: Execute single day's trading with retry logic +- `run_date_range(init_date, end_date)`: Process all weekdays in range +- `get_trading_dates()`: Resume from last date in position.jsonl +- `register_agent()`: Create initial position file with $10,000 cash + +**Adding Custom Agents:** +1. Create new class inheriting from `BaseAgent` +2. Add to `AGENT_REGISTRY` in `main.py`: + ```python + "CustomAgent": { + "module": "agent.custom.custom_agent", + "class": "CustomAgent" + } + ``` +3. Set `"agent_type": "CustomAgent"` in config JSON + +### System Prompt Construction + +**Dynamic prompt generation** (`prompts/agent_prompt.py`): +- `get_agent_system_prompt()` builds prompt with: + - Current date + - Yesterday's closing positions + - Yesterday's closing prices + - Today's buy prices + - Yesterday's profit/loss +- AI agent must output `` to end trading session + +### Anti-Look-Ahead Controls + +**Data access restrictions:** +- Price data: Only returns data for `date <= TODAY_DATE` +- Search results: News filtered by publication date +- All tools enforce temporal boundaries via `TODAY_DATE` from `runtime_env.json` + +## Configuration File Format + +```json +{ + "agent_type": "BaseAgent", + "date_range": { + "init_date": "2025-01-01", + "end_date": "2025-01-31" + }, + "models": [ + { + "name": "model-display-name", + "basemodel": "provider/model-id", + "signature": "unique-identifier", + "enabled": true, + "openai_base_url": "optional-override", + "openai_api_key": "optional-override" + } + ], + "agent_config": { + "max_steps": 30, // Max reasoning iterations per day + "max_retries": 3, // Retry attempts on failure + "base_delay": 1.0, // Base retry delay (seconds) + "initial_cash": 10000.0 + }, + "log_config": { + "log_path": "./data/agent_data" + } +} +``` + +## Data Formats + +**Position Record** (`position.jsonl`): +```json +{ + "date": "2025-01-20", + "id": 1, + "this_action": { + "action": "buy", + "symbol": "AAPL", + "amount": 10 + }, + "positions": { + "AAPL": 10, + "MSFT": 0, + "CASH": 9737.6 + } +} +``` + +**Price Data** (`merged.jsonl`): +```json +{ + "Meta Data": { + "2. Symbol": "AAPL", + "3. Last Refreshed": "2025-01-20" + }, + "Time Series (Daily)": { + "2025-01-20": { + "1. buy price": "255.8850", + "2. high": "264.3750", + "3. low": "255.6300", + "4. sell price": "262.2400", + "5. volume": "90483029" + } + } +} +``` + +## Important Implementation Details + +**Trading Day Logic:** +- Only weekdays (Monday-Friday) are processed +- `get_trading_dates()` automatically resumes from last date in `position.jsonl` +- Skips days already processed (idempotent) + +**Error Handling:** +- All async operations use `_ainvoke_with_retry()` with exponential backoff +- MCP service failures raise detailed error messages with troubleshooting hints +- Missing API keys halt startup with clear error messages + +**Tool Message Extraction:** +- `extract_conversation(response, "final")`: Get AI's final answer +- `extract_tool_messages(response)`: Get all tool results +- Handles both dict and object-based message formats + +**Logging:** +- Each trading day creates `log/[date]/log.jsonl` +- Logs include timestamps, signature, and all message exchanges +- Position updates append to single `position/position.jsonl` + +## Testing Changes + +When modifying agent behavior or adding tools: +1. Create test config with short date range (2-3 days) +2. Set `max_steps` low (e.g., 10) to iterate faster +3. Check logs in `data/agent_data/[signature]/log/[date]/` +4. Verify position updates in `position/position.jsonl` +5. Use `main.sh` only for full end-to-end testing + +## Common Issues + +**MCP Services Not Running:** +- Error: "Failed to initialize MCP client" +- Fix: `cd agent_tools && python start_mcp_services.py` +- Verify ports not already in use: `lsof -i :8000-8003` + +**Missing Price Data:** +- Ensure `data/merged.jsonl` exists +- Run `cd data && python get_daily_price.py && python merge_jsonl.py` +- Check Alpha Vantage API key is valid + +**Runtime Config Issues:** +- Set `RUNTIME_ENV_PATH` to absolute path in `.env` +- Ensure directory is writable +- File gets created automatically on first run + +**Agent Doesn't Stop Trading:** +- Agent must output `` within `max_steps` +- Increase `max_steps` if agent needs more reasoning time +- Check `log.jsonl` for errors preventing completion \ No newline at end of file