Read documents via Markdown conversion instead of images and python-docx
Step 2 rewrite. Steps 3-5, the JSON envelope, and the skill name are unchanged, so the five dependent skills and the 62 existing extraction artifacts keep working. - .docx path was broken: it required python3 + python-docx, and there is no python3 on the target machine. Removed the script and its test. - PDFs were read as rendered page images at 2.6-7.3x the token cost of the equivalent Markdown, widening with page count. - Adds xlsx, pptx, odt, ods, odp, rtf, epub, csv and legacy doc/xls/ppt, none of which were previously readable. Spreadsheets in particular arrive as real Markdown tables with shared strings and number formats resolved. Conversion and its fallback chain are delegated to the convert-documents-to-markdown skill rather than duplicated here.
This commit is contained in:
Bill committed
1 parent
eec37670a5
commit
4cecee39b5
3 files changed
+33
-214
No files matched your search
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: tabular-extract
|
||||
description: Extract structured data from document collections into tabular format. Reads PDFs, DOCX, TXT, MD, and JSON files from local paths, infers extraction columns from natural language descriptions, and outputs a markdown table plus a JSON file with values, confidence scores, supporting quotes, and reasoning. Use when the user asks to extract structured data from documents, turn documents into a spreadsheet or table, review or compare multiple documents side by side, or pull specific fields from a set of files.
|
||||
description: Extract structured data from document collections into tabular format. Reads PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, CSV, TXT, MD, and JSON files from local paths by converting them to Markdown first, infers extraction columns from natural language descriptions, and outputs a markdown table plus a JSON file with values, confidence scores, supporting quotes, and reasoning. Use when the user asks to extract structured data from documents, turn documents into a spreadsheet or table, review or compare multiple documents side by side, or pull specific fields from a set of files.
|
||||
---
|
||||
|
||||
# Tabular Extract
|
||||
@@ -22,22 +22,46 @@ This is a rigid, sequential pipeline. Execute every step in order.
|
||||
Glob the user-provided path for supported file types:
|
||||
|
||||
```bash
|
||||
**/*.pdf **/*.docx **/*.txt **/*.md **/*.json
|
||||
**/*.pdf **/*.docx **/*.doc **/*.odt **/*.rtf **/*.epub
|
||||
**/*.xlsx **/*.xls **/*.ods **/*.csv
|
||||
**/*.pptx **/*.ppt **/*.odp
|
||||
**/*.txt **/*.md **/*.json
|
||||
```
|
||||
|
||||
If a `.md` already exists beside a binary source from an earlier run, prefer it and skip
|
||||
reconversion.
|
||||
|
||||
Display the file list and count. Ask the user to confirm before proceeding.
|
||||
If no supported files are found, tell the user and stop.
|
||||
|
||||
## Step 2: Read Documents
|
||||
|
||||
Convert each file to text based on its type:
|
||||
**Convert every binary document to Markdown first, then read the Markdown.** Do not read
|
||||
PDFs or Office files directly — rendering pages as images costs 2.6–7.3× the tokens of the
|
||||
equivalent Markdown, and the gap widens with page count.
|
||||
|
||||
| Format | Method |
|
||||
|--------|--------|
|
||||
| .pdf | Use the Read tool with `pages` parameter for large files (>10 pages: read in chunks of 20 pages) |
|
||||
| .docx | Run: `python3 ~/.claude/skills/tabular-extract/scripts/convert_docx.py <filepath>` (requires `pip install python-docx`) |
|
||||
| .txt, .md | Use the Read tool directly |
|
||||
| .json | Use the Read tool directly |
|
||||
| .txt, .md, .json | Use the Read tool directly — already text |
|
||||
| everything else | Convert to Markdown with the `convert-documents-to-markdown` skill, write it next to the source with `-o`, then Read that |
|
||||
|
||||
```bash
|
||||
npx -y @firecrawl/anydoc <file> -o <file>.md
|
||||
```
|
||||
|
||||
Invoke the `convert-documents-to-markdown` skill rather than reimplementing this. It owns
|
||||
the fallback chain — a PDF that fails to parse is retried with a text extractor, and a
|
||||
genuinely scanned document is routed to OCR. A failed conversion does **not** mean the
|
||||
document is unreadable, so follow that skill's error-routing rather than skipping on the
|
||||
first non-zero exit.
|
||||
|
||||
Converting first also widens what this pipeline accepts. Beyond PDF and DOCX it handles
|
||||
`.xlsx`, `.pptx`, `.odt`, `.ods`, `.odp`, `.rtf`, `.epub`, `.csv`, and the legacy `.doc`,
|
||||
`.xls`, `.ppt` — spreadsheets in particular arrive as real Markdown tables with shared
|
||||
strings resolved and number formats applied, which raw XML extraction does not give you.
|
||||
|
||||
Keep the generated `.md` files. They are the reusable artifact: greppable by later agents,
|
||||
and re-runnable for a different schema without re-reading the sources.
|
||||
|
||||
If a file fails to convert, log it as skipped and continue with remaining files. Do not stop the pipeline.
|
||||
|
||||
@@ -94,8 +118,8 @@ See `references/extraction-guide.md` for detailed type handling, confidence crit
|
||||
|
||||
## Error Handling
|
||||
|
||||
- **Missing python-docx**: Print install command `pip install python-docx` and ask user to install
|
||||
- **Conversion fails**: Follow the `convert-documents-to-markdown` skill's error routing before giving up — most failures are recoverable by a different reader, and only an explicit OCR message means the document is truly image-only. Skip only after that chain is exhausted.
|
||||
- **Unreadable file**: Skip file, record in skipped list, continue pipeline
|
||||
- **Large PDF (>10 pages)**: Read in 20-page chunks, concatenate text
|
||||
- **Large document**: Convert to Markdown with `-o` and read the sections you need, rather than pulling the whole file into context
|
||||
- **No files found**: Inform user and stop
|
||||
- **User cancels at confirmation**: Stop gracefully
|
||||
@@ -1,83 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Convert a DOCX file to plain text.
|
||||
|
||||
Extracts text from paragraphs and tables.
|
||||
Outputs to stdout for piping into other tools.
|
||||
|
||||
Usage:
|
||||
convert_docx.py <path-to-file.docx>
|
||||
|
||||
Requires:
|
||||
pip install python-docx
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
|
||||
def convert_docx(filepath: str) -> str:
|
||||
"""Extract text from a DOCX file, including paragraphs and tables."""
|
||||
try:
|
||||
from docx import Document
|
||||
except ImportError:
|
||||
print(
|
||||
"Error: python-docx is not installed.\n"
|
||||
"Install it with: pip install python-docx",
|
||||
file=sys.stderr
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
doc = Document(filepath)
|
||||
parts = []
|
||||
|
||||
for element in doc.element.body:
|
||||
tag = element.tag.split("}")[-1] if "}" in element.tag else element.tag
|
||||
|
||||
if tag == "p":
|
||||
# Paragraph
|
||||
for para in doc.paragraphs:
|
||||
if para._element is element:
|
||||
text = para.text.strip()
|
||||
if text:
|
||||
parts.append(text)
|
||||
break
|
||||
|
||||
elif tag == "tbl":
|
||||
# Table
|
||||
for table in doc.tables:
|
||||
if table._element is element:
|
||||
for row in table.rows:
|
||||
cells = [cell.text.strip() for cell in row.cells]
|
||||
parts.append(" | ".join(cells))
|
||||
parts.append("") # blank line after table
|
||||
break
|
||||
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 2:
|
||||
print("Usage: convert_docx.py <path-to-file.docx>", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
filepath = sys.argv[1]
|
||||
|
||||
if not os.path.exists(filepath):
|
||||
print(f"Error: File not found: {filepath}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if not filepath.lower().endswith(".docx"):
|
||||
print(f"Error: Not a .docx file: {filepath}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
text = convert_docx(filepath)
|
||||
print(text)
|
||||
except Exception as e:
|
||||
print(f"Error converting file: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,122 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for convert_docx.py"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
SCRIPT = os.path.join(os.path.dirname(__file__), "convert_docx.py")
|
||||
|
||||
|
||||
def test_missing_argument():
|
||||
"""Script should print usage and exit 1 when no args given."""
|
||||
result = subprocess.run(
|
||||
[sys.executable, SCRIPT],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
assert result.returncode == 1
|
||||
assert "Usage:" in result.stderr
|
||||
|
||||
|
||||
def test_nonexistent_file():
|
||||
"""Script should error on a file that doesn't exist."""
|
||||
result = subprocess.run(
|
||||
[sys.executable, SCRIPT, "/tmp/nonexistent_file_abc123.docx"],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
assert result.returncode == 1
|
||||
assert "Error" in result.stderr or "not found" in result.stderr.lower()
|
||||
|
||||
|
||||
def test_non_docx_file():
|
||||
"""Script should error on a non-DOCX file."""
|
||||
with tempfile.NamedTemporaryFile(suffix=".txt", delete=False) as f:
|
||||
f.write(b"hello world")
|
||||
f.flush()
|
||||
result = subprocess.run(
|
||||
[sys.executable, SCRIPT, f.name],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
os.unlink(f.name)
|
||||
assert result.returncode == 1
|
||||
|
||||
|
||||
def test_valid_docx():
|
||||
"""Script should extract text from a valid DOCX file."""
|
||||
try:
|
||||
from docx import Document
|
||||
except ImportError:
|
||||
print("SKIP: python-docx not installed")
|
||||
return
|
||||
|
||||
doc = Document()
|
||||
doc.add_paragraph("Hello from test document")
|
||||
doc.add_paragraph("Second paragraph here")
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".docx", delete=False) as f:
|
||||
doc.save(f.name)
|
||||
result = subprocess.run(
|
||||
[sys.executable, SCRIPT, f.name],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
os.unlink(f.name)
|
||||
|
||||
assert result.returncode == 0
|
||||
assert "Hello from test document" in result.stdout
|
||||
assert "Second paragraph here" in result.stdout
|
||||
|
||||
|
||||
def test_docx_with_table():
|
||||
"""Script should extract table content from a DOCX file."""
|
||||
try:
|
||||
from docx import Document
|
||||
except ImportError:
|
||||
print("SKIP: python-docx not installed")
|
||||
return
|
||||
|
||||
doc = Document()
|
||||
doc.add_paragraph("Before table")
|
||||
table = doc.add_table(rows=2, cols=2)
|
||||
table.cell(0, 0).text = "Header1"
|
||||
table.cell(0, 1).text = "Header2"
|
||||
table.cell(1, 0).text = "Value1"
|
||||
table.cell(1, 1).text = "Value2"
|
||||
doc.add_paragraph("After table")
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".docx", delete=False) as f:
|
||||
doc.save(f.name)
|
||||
result = subprocess.run(
|
||||
[sys.executable, SCRIPT, f.name],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
os.unlink(f.name)
|
||||
|
||||
assert result.returncode == 0
|
||||
assert "Header1" in result.stdout
|
||||
assert "Value1" in result.stdout
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
tests = [
|
||||
test_missing_argument,
|
||||
test_nonexistent_file,
|
||||
test_non_docx_file,
|
||||
test_valid_docx,
|
||||
test_docx_with_table,
|
||||
]
|
||||
passed = 0
|
||||
failed = 0
|
||||
for test in tests:
|
||||
try:
|
||||
test()
|
||||
print(f" PASS: {test.__name__}")
|
||||
passed += 1
|
||||
except AssertionError as e:
|
||||
print(f" FAIL: {test.__name__} - {e}")
|
||||
failed += 1
|
||||
except Exception as e:
|
||||
print(f" ERROR: {test.__name__} - {e}")
|
||||
failed += 1
|
||||
print(f"\n{passed} passed, {failed} failed")
|
||||
sys.exit(1 if failed else 0)
|
||||
Reference in new issue
Block a user