Changed merge_jsonl.py to use os.getcwd() instead of os.path.dirname(__file__)
to ensure merged.jsonl is written to the working directory where data files exist,
not to the script's installation directory.
Root cause:
- Dockerfile copies scripts to /app/scripts/ for volume compatibility
- entrypoint.sh runs: cd /app/data && python /app/scripts/merge_jsonl.py
- Old logic used script directory (/app/scripts/), ignoring working directory
- This caused merged.jsonl to be created in /app/scripts/ instead of /app/data/
- Since /app/data/ is volume-mounted, merged file was not visible to host
Solution:
- Scripts now respect current working directory (Unix philosophy)
- Works correctly with volume mounts and script relocation
- Tested in both local and Docker directory structure scenarios
Fixes the issue where merged.jsonl was missing from mounted data volume.
Streamline the data preparation workflow by having get_daily_price.py
automatically invoke merge_jsonl.py after fetching all stock prices.
Changes:
- Modified get_daily_price.py to call merge_jsonl.py automatically
- Updated entrypoint.sh to remove redundant merge_jsonl.py call
- Updated main.sh to remove redundant merge_jsonl.py call
- Fixed import order for linting compliance
Benefits:
- Single command now handles both fetching and merging
- Ensures merged.jsonl is always created after price updates
- Simplifies Docker container startup process
- Prevents missing merged.jsonl errors in production
Replace verbose JSON logging with concise status messages:
- Success: '✓ Fetched SYMBOL'
- Error: '⚠️ SYMBOL: API rate limit or error - [message]'
Prevents logs from being flooded with full JSON responses
for 100+ stock symbols during startup.
