Implement skip status tracking to fix jobs hanging when dates are
filtered out. Jobs now properly complete when all model-days reach
terminal states (completed/failed/skipped).
Changes:
- database.py: Add 'skipped' status to job_details CHECK constraint
- job_manager.py: Update completion logic to count skipped as done
- job_manager.py: Add skipped count to progress tracking
- simulation_worker.py: Implement skip tracking with per-model granularity
- simulation_worker.py: Add _filter_completed_dates_with_tracking()
- simulation_worker.py: Add _mark_skipped_dates()
- simulation_worker.py: Update _prepare_data() to use skip tracking
- simulation_worker.py: Improve warning messages to distinguish skip types
Skip reasons:
- "Already completed" - Position data exists from previous job
- "Incomplete price data" - Missing prices (weekends/holidays/future)
The implementation correctly handles multi-model scenarios where different
models have different completion states for the same date.
Remove complex table recreation logic since the server hasn't been
deployed yet. For existing databases, simply delete and recreate.
The dev database is already recreated on startup by design.
Co-Authored-By: Claude <noreply@anthropic.com>
Critical fixes identified in code review:
1. Add warnings column migration to _migrate_schema()
- Checks if warnings column exists in jobs table
- Adds column via ALTER TABLE if missing
- Ensures existing databases get new column on upgrade
2. Document CHECK constraint limitation
- Added docstring explaining ALTER TABLE cannot add CHECK constraints
- Notes that "downloading_data" status requires fresh DB or manual migration
3. Add comprehensive migration tests
- test_migration_adds_warnings_column: Verifies warnings column migration
- test_migration_adds_simulation_run_id_column: Tests existing migration
- Both tests include cleanup to prevent cross-test contamination
4. Update test fixtures and expectations
- Updated clean_db fixture to delete from all 9 tables
- Fixed table count assertions (6 -> 9 tables)
- Updated expected columns in schema tests
All 21 database tests now pass.
Add support for:
- downloading_data job status for visibility during data prep
- warnings TEXT column for storing job-level warnings (JSON array)
Co-Authored-By: Claude <noreply@anthropic.com>
Add automatic schema migration to handle existing databases that don't
have the simulation_run_id column in the positions table.
Problem:
- v0.3.0-alpha.3 databases lack simulation_run_id column
- CREATE TABLE IF NOT EXISTS doesn't add new columns to existing tables
- Index creation fails with "no such column: simulation_run_id"
Solution:
- Add _migrate_schema() function to detect and migrate old schemas
- Check if positions table exists and inspect its columns
- ALTER TABLE to add simulation_run_id if missing
- Run migration before creating indexes
This allows seamless upgrades from alpha.3 to alpha.4 without manual
database deletion or migration scripts.
Fixes docker compose startup error:
sqlite3.OperationalError: no such column: simulation_run_id
Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 1 of v0.3.0 date range and on-demand download implementation.
Database changes:
- Add price_data table (OHLCV data, replaces merged.jsonl)
- Add price_data_coverage table (track downloaded date ranges)
- Add simulation_runs table (soft delete support)
- Add simulation_run_id to positions table
- Add comprehensive indexes for new tables
New modules:
- api/price_data_manager.py - Priority-based download manager
- Coverage gap detection
- Smart download prioritization (maximize date completion)
- Rate limit handling with retry logic
- Alpha Vantage integration
Configuration:
- configs/nasdaq100_symbols.json - NASDAQ 100 constituent list
Next steps (not yet implemented):
- Migration script for merged.jsonl -> price_data
- Update API models (start_date/end_date)
- Update tools/price_tools.py to read from database
- Simulation run tracking implementation
- API integration
- Tests and documentation
This is work in progress for the v0.3.0 release.
