docs: add database-only position tracking design

Created comprehensive design document addressing: - ContextInjector initialization timing issues - Migration from file-based to database-only position tracking - Complete data flow and integration strategy - Testing and validation approach Design resolves two critical simulation failures: 1. ContextInjector receiving None values for trade tool parameters 2. FileNotFoundError when accessing position.jsonl files Ready for implementation.
2026-04-01 17:17:24 -04:00 · 2025-11-02 22:11:00 -05:00
parent fcfdf36c1c
commit 7a734d265b
1 changed files with 476 additions and 0 deletions
--- a/docs/plans/2025-02-11-database-position-tracking-design.md
+++ b/docs/plans/2025-02-11-database-position-tracking-design.md
@@ -0,0 +1,476 @@
+# Database-Only Position Tracking Design
+
+**Date:** 2025-02-11
+**Status:** Approved
+**Version:** 1.0
+
+## Problem Statement
+
+Two critical issues prevent simulations from running:
+
+1. **ContextInjector receives None values**: The ContextInjector shows `{'signature': None, 'today_date': None, 'job_id': None, 'session_id': None}` when injecting parameters into trade tool calls, causing trade validation to fail.
+
+2. **File-based position tracking still in use**: System prompt builder and no-trade handler attempt to read/write position.jsonl files that no longer exist after SQLite migration.
+
+## Root Cause Analysis
+
+### Issue 1: ContextInjector Initialization Timing
+
+**Problem Chain:**
+- `BaseAgent.__init__()` creates `ContextInjector` with `self.init_date`
+- `init_date` is the START of simulation date range (e.g., "2025-10-13"), not current trading day ("2025-10-01")
+- Runtime config contains correct values (`TODAY_DATE="2025-10-01"`, `SIGNATURE="gpt-5"`, `JOB_ID="dc488e87..."`), but BaseAgent doesn't use them during initialization
+- ContextInjector is created before the trading session, so it doesn't know the correct date
+
+**Evidence:**
+```
+ai-trader-app  | [ContextInjector] Tool: buy, Args after injection: {'symbol': 'MSFT', 'amount': 1, 'signature': None, 'today_date': None, 'job_id': None, 'session_id': None}
+```
+
+### Issue 2: Mixed Storage Architecture
+
+**Problem Chain:**
+- Trade tools (tool_trade.py) query/write to SQLite database
+- System prompt builder calls `get_today_init_position()` which reads position.jsonl files
+- No-trade handler calls `add_no_trade_record()` which writes to position.jsonl files
+- Files don't exist because we migrated to database-only storage
+
+**Evidence:**
+```
+FileNotFoundError: [Errno 2] No such file or directory: '/app/data/agent_data/gpt-5/position/position.jsonl'
+```
+
+## Design Solution
+
+### Architecture Principles
+
+1. **Database-only position storage**: All position queries and writes go through SQLite
+2. **Lazy context injection**: Create ContextInjector after runtime config is written and session is created
+3. **Real-time database queries**: System prompt builder queries database directly, no file caching
+4. **Clean initialization order**: Config → Database → Agent → Context → Session
+
+### Component Changes
+
+#### 1. ContextInjector Lifecycle Refactor
+
+**BaseAgent Changes:**
+
+Remove ContextInjector creation from `__init__()`:
+```python
+# OLD (in __init__)
+self.context_injector = ContextInjector(
+    signature=self.signature,
+    today_date=self.init_date,  # WRONG: uses start date
+    job_id=job_id
+)
+self.client = MultiServerMCPClient(
+    self.mcp_config,
+    tool_interceptors=[self.context_injector]
+)
+
+# NEW (in __init__)
+self.context_injector = None
+self.client = MultiServerMCPClient(
+    self.mcp_config,
+    tool_interceptors=[]  # Empty initially
+)
+```
+
+Add new method `set_context()`:
+```python
+def set_context(self, context_injector: ContextInjector) -> None:
+    """Inject ContextInjector after initialization.
+
+    Args:
+        context_injector: Configured ContextInjector instance
+    """
+    self.context_injector = context_injector
+    self.client.add_interceptor(context_injector)
+```
+
+**ModelDayExecutor Changes:**
+
+Create and inject ContextInjector after agent initialization:
+```python
+async def execute_async(self) -> Dict[str, Any]:
+    # ... create session, initialize position ...
+
+    # Set RUNTIME_ENV_PATH
+    os.environ["RUNTIME_ENV_PATH"] = self.runtime_config_path
+
+    # Initialize agent (without context)
+    agent = await self._initialize_agent()
+
+    # Create context injector with correct values
+    context_injector = ContextInjector(
+        signature=self.model_sig,
+        today_date=self.date,  # CORRECT: current trading day
+        job_id=self.job_id,
+        session_id=session_id
+    )
+
+    # Inject context into agent
+    agent.set_context(context_injector)
+
+    # Run trading session
+    session_result = await agent.run_trading_session(self.date)
+```
+
+#### 2. Database Position Query Functions
+
+**New Functions (tools/price_tools.py):**
+
+```python
+def get_today_init_position_from_db(
+    today_date: str,
+    modelname: str,
+    job_id: str
+) -> Dict[str, float]:
+    """
+    Query yesterday's position from database.
+
+    Args:
+        today_date: Current trading date (YYYY-MM-DD)
+        modelname: Model signature
+        job_id: Job UUID
+
+    Returns:
+        Position dict: {"AAPL": 50, "MSFT": 30, "CASH": 5000.0}
+        If no position exists: {"CASH": 10000.0} (initial cash)
+    """
+    from tools.deployment_config import get_db_path
+    from api.database import get_db_connection
+
+    db_path = get_db_path()
+    conn = get_db_connection(db_path)
+    cursor = conn.cursor()
+
+    try:
+        # Get most recent position before today
+        cursor.execute("""
+            SELECT p.id, p.cash
+            FROM positions p
+            WHERE p.job_id = ? AND p.model = ? AND p.date < ?
+            ORDER BY p.date DESC, p.action_id DESC
+            LIMIT 1
+        """, (job_id, modelname, today_date))
+
+        row = cursor.fetchone()
+
+        if not row:
+            # First day - return initial cash
+            return {"CASH": 10000.0}  # TODO: Read from config
+
+        position_id, cash = row
+        position_dict = {"CASH": cash}
+
+        # Get holdings for this position
+        cursor.execute("""
+            SELECT symbol, quantity
+            FROM holdings
+            WHERE position_id = ?
+        """, (position_id,))
+
+        for symbol, quantity in cursor.fetchall():
+            position_dict[symbol] = quantity
+
+        return position_dict
+
+    finally:
+        conn.close()
+
+
+def add_no_trade_record_to_db(
+    today_date: str,
+    modelname: str,
+    job_id: str,
+    session_id: int
+) -> None:
+    """
+    Create no-trade position record in database.
+
+    Args:
+        today_date: Current trading date (YYYY-MM-DD)
+        modelname: Model signature
+        job_id: Job UUID
+        session_id: Trading session ID
+    """
+    from tools.deployment_config import get_db_path
+    from api.database import get_db_connection
+    from agent_tools.tool_trade import get_current_position_from_db
+    from datetime import datetime
+
+    db_path = get_db_path()
+    conn = get_db_connection(db_path)
+    cursor = conn.cursor()
+
+    try:
+        # Get current position
+        current_position, next_action_id = get_current_position_from_db(
+            job_id, modelname, today_date
+        )
+
+        # Calculate portfolio value
+        # (Reuse logic from tool_trade.py)
+        cash = current_position.get("CASH", 0.0)
+        portfolio_value = cash
+
+        # Add stock values
+        for symbol, qty in current_position.items():
+            if symbol != "CASH":
+                try:
+                    from tools.price_tools import get_open_prices
+                    price = get_open_prices(today_date, [symbol])[f'{symbol}_price']
+                    portfolio_value += qty * price
+                except KeyError:
+                    pass
+
+        # Get previous value for P&L
+        cursor.execute("""
+            SELECT portfolio_value
+            FROM positions
+            WHERE job_id = ? AND model = ? AND date < ?
+            ORDER BY date DESC, action_id DESC
+            LIMIT 1
+        """, (job_id, modelname, today_date))
+
+        row = cursor.fetchone()
+        previous_value = row[0] if row else 10000.0
+
+        daily_profit = portfolio_value - previous_value
+        daily_return_pct = (daily_profit / previous_value * 100) if previous_value > 0 else 0
+
+        # Insert position record
+        created_at = datetime.utcnow().isoformat() + "Z"
+
+        cursor.execute("""
+            INSERT INTO positions (
+                job_id, date, model, action_id, action_type,
+                cash, portfolio_value, daily_profit, daily_return_pct,
+                session_id, created_at
+            )
+            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+        """, (
+            job_id, today_date, modelname, next_action_id, "no_trade",
+            cash, portfolio_value, daily_profit, daily_return_pct,
+            session_id, created_at
+        ))
+
+        position_id = cursor.lastrowid
+
+        # Insert holdings (unchanged from previous position)
+        for symbol, qty in current_position.items():
+            if symbol != "CASH":
+                cursor.execute("""
+                    INSERT INTO holdings (position_id, symbol, quantity)
+                    VALUES (?, ?, ?)
+                """, (position_id, symbol, qty))
+
+        conn.commit()
+
+    except Exception as e:
+        conn.rollback()
+        raise
+    finally:
+        conn.close()
+```
+
+#### 3. System Prompt Builder Updates
+
+**Modified Function (prompts/agent_prompt.py):**
+
+```python
+def get_agent_system_prompt(today_date: str, signature: str) -> str:
+    """Build system prompt with database position queries."""
+    from tools.general_tools import get_config_value
+
+    print(f"signature: {signature}")
+    print(f"today_date: {today_date}")
+
+    # Get job_id from runtime config
+    job_id = get_config_value("JOB_ID")
+    if not job_id:
+        raise ValueError("JOB_ID not found in runtime config")
+
+    # Query database for yesterday's position
+    today_init_position = get_today_init_position_from_db(
+        today_date, signature, job_id
+    )
+
+    # Get prices (unchanged)
+    yesterday_buy_prices, yesterday_sell_prices = get_yesterday_open_and_close_price(
+        today_date, all_nasdaq_100_symbols
+    )
+    today_buy_price = get_open_prices(today_date, all_nasdaq_100_symbols)
+    yesterday_profit = get_yesterday_profit(
+        today_date, yesterday_buy_prices, yesterday_sell_prices, today_init_position
+    )
+
+    return agent_system_prompt.format(
+        date=today_date,
+        positions=today_init_position,
+        STOP_SIGNAL=STOP_SIGNAL,
+        yesterday_close_price=yesterday_sell_prices,
+        today_buy_price=today_buy_price,
+        yesterday_profit=yesterday_profit
+    )
+```
+
+#### 4. No-Trade Handler Updates
+
+**Modified Method (agent/base_agent/base_agent.py):**
+
+```python
+async def _handle_trading_result(self, today_date: str) -> None:
+    """Handle trading results with database writes."""
+    from tools.general_tools import get_config_value
+    from tools.price_tools import add_no_trade_record_to_db
+
+    if_trade = get_config_value("IF_TRADE")
+
+    if if_trade:
+        write_config_value("IF_TRADE", False)
+        print("✅ Trading completed")
+    else:
+        print("📊 No trading, maintaining positions")
+
+        # Get context from runtime config
+        job_id = get_config_value("JOB_ID")
+        session_id = self.context_injector.session_id if self.context_injector else None
+
+        if not job_id or not session_id:
+            raise ValueError("Missing JOB_ID or session_id for no-trade record")
+
+        # Write no-trade record to database
+        add_no_trade_record_to_db(
+            today_date,
+            self.signature,
+            job_id,
+            session_id
+        )
+
+        write_config_value("IF_TRADE", False)
+```
+
+### Data Flow Summary
+
+**Complete Execution Sequence:**
+
+1. `ModelDayExecutor.__init__()`:
+   - Create runtime config file with TODAY_DATE, SIGNATURE, JOB_ID
+
+2. `ModelDayExecutor.execute_async()`:
+   - Create trading_sessions record → get session_id
+   - Initialize starting position (if first day)
+   - Set RUNTIME_ENV_PATH environment variable
+   - Initialize agent (without ContextInjector)
+   - Create ContextInjector(date, model_sig, job_id, session_id)
+   - Call agent.set_context(context_injector)
+   - Run trading session
+
+3. `BaseAgent.run_trading_session()`:
+   - Build system prompt → queries database for yesterday's position
+   - AI agent analyzes and decides
+   - Calls buy/sell tools → ContextInjector injects parameters
+   - Trade tools write to database
+   - If no trade: add_no_trade_record_to_db()
+
+4. Position Query Flow:
+   - System prompt needs yesterday's position
+   - `get_today_init_position_from_db(today_date, signature, job_id)`
+   - Query: `SELECT positions + holdings WHERE job_id=? AND model=? AND date<? ORDER BY date DESC, action_id DESC LIMIT 1`
+   - Reconstruct position dict from results
+   - Return to system prompt builder
+
+### Testing Strategy
+
+**Critical Test Cases:**
+
+1. **First Trading Day**
+   - No previous position in database
+   - Returns `{"CASH": 10000.0}`
+   - System prompt shows available cash
+   - Initial position created with action_id=0
+
+2. **Subsequent Trading Days**
+   - Query finds previous position
+   - System prompt shows yesterday's holdings
+   - Action_id increments properly
+
+3. **No-Trade Days**
+   - Agent outputs `<FINISH_SIGNAL>` without trading
+   - `add_no_trade_record_to_db()` creates record
+   - Holdings unchanged
+   - Portfolio value calculated
+
+4. **ContextInjector Values**
+   - All parameters non-None
+   - Debug log shows correct injection
+   - Trade tools validate successfully
+
+**Edge Cases:**
+
+- Multiple models, same job (different signatures)
+- Date gaps (weekends) - query finds Friday on Monday
+- Mid-simulation restart - resumes from last position
+- Empty holdings (only CASH)
+
+**Validation Points:**
+
+- Log ContextInjector values at injection
+- Log database query results
+- Verify initial position created
+- Check session_id links positions
+
+## Implementation Checklist
+
+### Phase 1: ContextInjector Refactor
+- [ ] Remove ContextInjector creation from BaseAgent.__init__()
+- [ ] Add BaseAgent.set_context() method
+- [ ] Update ModelDayExecutor to create and inject ContextInjector
+- [ ] Add debug logging for injected values
+
+### Phase 2: Database Position Functions
+- [ ] Implement get_today_init_position_from_db()
+- [ ] Implement add_no_trade_record_to_db()
+- [ ] Add database error handling
+- [ ] Add logging for query results
+
+### Phase 3: Integration
+- [ ] Update get_agent_system_prompt() to use database queries
+- [ ] Update _handle_trading_result() to use database writes
+- [ ] Remove/deprecate old file-based functions
+- [ ] Test first trading day scenario
+- [ ] Test subsequent trading days
+- [ ] Test no-trade scenario
+
+### Phase 4: Validation
+- [ ] Run full simulation and verify ContextInjector logs
+- [ ] Verify initial cash appears in system prompt
+- [ ] Verify trades execute successfully
+- [ ] Verify no-trade records created
+- [ ] Check database for correct position records
+
+## Rollback Plan
+
+If issues arise:
+1. Revert ContextInjector changes (keep in __init__)
+2. Temporarily pass correct date via environment variable
+3. Keep file-based functions as fallback
+4. Debug database queries in isolation
+
+## Success Criteria
+
+1. ContextInjector logs show all non-None values
+2. System prompt displays initial $10,000 cash
+3. Trade tools successfully execute buy/sell operations
+4. No FileNotFoundError exceptions
+5. Database contains correct position records
+6. AI agent can complete full trading day
+
+## Notes
+
+- File-based functions marked as deprecated but not removed (backward compatibility)
+- Database queries use deployment_config for automatic prod/dev resolution
+- Initial cash value should eventually be read from config, not hardcoded
+- Consider adding database connection pooling for performance