xer-mcp/specs/002-direct-db-access/research.md

Research: Direct Database Access for Scripts

Date: 2026-01-08 Branch: 002-direct-db-access

Research Topics

1. SQLite File-Based vs In-Memory Database

Decision: Support both file-based and in-memory databases through the same DatabaseManager interface.

Rationale:

• File-based SQLite allows external scripts to query data directly without MCP overhead
• In-memory remains the default for backward compatibility with existing tools
• SQLite's file format is universally readable (Python sqlite3, DBeaver, sqlitebrowser, etc.)
• Single connection string change (:memory: vs file path) switches modes

Alternatives Considered:

• Separate database manager for files: Rejected - unnecessary duplication; SQLite handles both modes identically
• Export to CSV/JSON: Rejected - loses relational structure; no query capability
• Network database (PostgreSQL): Rejected - overkill for single-user local access; requires external server

2. Database File Location Strategy

Decision: Accept user-specified path, or default to a predictable location derived from the XER file path.

Rationale:

• User-specified path gives maximum flexibility for script integration
• Default path ({xer_file_directory}/{xer_basename}.sqlite) is predictable and colocated with source
• Absolute paths in responses eliminate ambiguity for scripts

Default Path Algorithm:

Input:  /path/to/schedule.xer
Output: /path/to/schedule.sqlite

Alternatives Considered:

• Temp directory only: Rejected - less predictable; may be cleaned up unexpectedly
• Fixed location (e.g., ~/.xer-mcp/db/): Rejected - less convenient; separates DB from source file
• Always require user to specify: Rejected - worse developer experience for common case

3. Atomic Write Strategy

Decision: Use SQLite's built-in transaction support; write to temp file and rename for atomicity.

Rationale:

• SQLite transactions ensure data integrity during writes
• Write-then-rename pattern prevents partial/corrupted files if process interrupted
• External scripts won't see incomplete data

Implementation:

1. Create database at {target_path}.tmp
2. Load all data within a transaction
3. Commit transaction
4. Rename {target_path}.tmp to {target_path} (atomic on POSIX)

Alternatives Considered:

• Direct write to final path: Rejected - risk of corruption on interruption
• Lock file mechanism: Rejected - SQLite already handles locking; adds complexity

4. Schema Introspection Approach

Decision: Query SQLite's sqlite_master table and PRAGMA table_info() for schema information.

Rationale:

• Standard SQLite introspection APIs - no custom metadata needed
• Returns actual schema, not just documentation
• Works with any SQLite database, even if created by different tools

Schema Information Returned:

• Table names
• Column names and types for each table
• Primary key information
• Foreign key relationships (via PRAGMA foreign_key_list())

Alternatives Considered:

• Hardcoded schema in response: Rejected - may drift from actual schema; not dynamic
• Separate metadata table: Rejected - adds complexity; standard introspection sufficient

5. Concurrent Access Handling

Decision: Use SQLite WAL (Write-Ahead Logging) mode for concurrent read access.

Rationale:

• WAL mode allows multiple readers while one writer operates
• Scripts can query while MCP server operates without blocking
• Minimal configuration change: PRAGMA journal_mode=WAL

Implementation:

• Enable WAL mode when creating file-based database
• In-memory databases don't need WAL (single connection)

Alternatives Considered:

• Reader/writer locks in application: Rejected - SQLite handles this natively
• Copy-on-read: Rejected - unnecessary with WAL mode

Technology Decisions Summary

Decision	Choice	Key Reason
Database format	SQLite file	Universal compatibility, no server needed
Default location	Same directory as XER file	Predictable, colocated
Atomicity	Temp file + rename	Prevents corruption
Schema info	SQLite introspection APIs	Dynamic, accurate
Concurrency	WAL mode	Multiple readers supported

Open Questions Resolved

All technical questions have been resolved through research. No clarifications needed.