name: reference-validation description: Validate references in an academic PDF. Phase 1 extracts text + parses refs. Phase 2 verifies EVERY ref individually via WebSearch (existence + topic relevance), confirms cited arXiv IDs via WebFetch, and flags venue conflicts. No API keys needed. Use when user says "validate references", "check citations", "verify bibliography". argument-hint: <pdf_file> allowed-tools: Bash(*), Read, Write, Edit, WebSearch, WebFetch

Reference Validation

Context: $ARGUMENTS

Output Location

All outputs go next to the PDF, in a ref_validation/ subdirectory derived from the PDF's own absolute path. Never write results to /tmp, the cwd, or the skill directory. The only exception is if the user explicitly names a different output directory.

Compute it once, deterministically, and reuse it everywhere:

PDF_FILE="$(cd "$(dirname "<input_pdf>")" && pwd)/$(basename "<input_pdf>")"  # absolute path
OUTDIR="$(dirname "$PDF_FILE")/ref_validation"

So for /papers/foo.pdf → outputs land in /papers/ref_validation/. With {name} = the PDF stem (foo):

| File | Purpose | |------|---------| | $OUTDIR/{name}_refs_text.txt | Raw references section text (Phase 1) | | $OUTDIR/{name}_meta.json | PDF metadata (Phase 1) | | $OUTDIR/{name}_extracted_refs.json | Structured reference list (Phase 1.2) | | $OUTDIR/{name}_results.json | Validation results accumulator (Phase 2) | | $OUTDIR/{name}_VALIDATION_REPORT.md | Final report (Phase 2.4) |

Startup: Optional Checks

At the very start, ask the user ONE question (use the AskUserQuestion tool if available, otherwise ask in plain text and wait):

Enable the evidence-URL reachability check? For each REAL reference I'll additionally WebFetch its evidence URL to confirm it resolves and actually matches the paper. More thorough, but noticeably slower and uses more tokens. (Default: OFF)

Record the answer as url_check_enabled (true/false). If the user skips or doesn't care, default to OFF.

The other two integrity checks are always on because they are cheap and low-noise:

arXiv-ID verification (Check 1b) — confirms a cited arXiv ID resolves to the right paper.
Venue consistency (Check 1c) — advisory flag when the cited venue clearly conflicts with search results.

Phase 1: Extract Text & Parse References

1.1 Extract references section from PDF

# Bundled scripts live next to this SKILL.md. Set SKILL_DIR to that absolute folder
# so the skill works wherever it is installed (e.g. ~/.claude/skills/reference-validation/).
SKILL_DIR="<absolute path of the directory containing this SKILL.md>"

# PDF_FILE and OUTDIR are the absolute paths computed in "Output Location" above.
mkdir -p "$OUTDIR"
python3 "$SKILL_DIR/run_pipeline.py" "$PDF_FILE" -o "$OUTDIR"

Requirements: pip install pymupdf (or pip install -r "$SKILL_DIR/requirements.txt"). No API keys needed. (run_pipeline.py writes only inside -o, so passing the PDF-derived $OUTDIR keeps everything next to the PDF.)

1.2 Parse structured references from text

Read $OUTDIR/{name}_refs_text.txt. It contains the raw bibliography section.

Extract structured references from this text into an array:

[{"title": "...", "authors": ["..."], "date": "...", "venue": "...", "arxiv_id": "..."}, ...]

Rules:

Extract EVERY reference — don't skip any
Preserve reference tags like [BS01] if present
If arxiv_id is mentioned (e.g., arXiv:XXXX.XXXXX), extract just the ID part
If a field is missing, use "N/A"
Save to $OUTDIR/{name}_extracted_refs.json

Phase 2: Validate with WebSearch

⚠️ CRITICAL: Validate EVERY reference individually.

Do NOT batch-approve unsearched references. Every single reference must have its own WebSearch query. If there are 55 refs, you run 55 searches. No exceptions.

2.1 Load references

Read $OUTDIR/{name}_extracted_refs.json.

2.2 For EACH reference, run these checks:

Two core checks (Existence + Topic Relevance) plus the integrity checks (1b/1c, and 1d if enabled):

Check 1 — Existence: WebSearch "{title}" {first_author_last_name} {venue} {date}

Judge:

🟢 REAL: Matching title + at least first-author last name match → record evidence URL
🔴 HALLUCINATION: Title found but completely different authors (attribution error), OR no results after 2 query attempts with different formulations
🟡 UNCERTAIN: Partial match, can't confirm or deny

Check 1b — arXiv ID verification (always on; only when the ref has an arxiv_id ≠ "N/A"):

WebFetch https://arxiv.org/abs/{arxiv_id} and compare the returned page title to the cited title.

arxiv_check: "match" — page loads and the title is the same paper
arxiv_check: "mismatch" — page loads but the title is a different paper → 🔴 wrong/typo'd arXiv ID; state it in reasoning
arxiv_check: "unreachable" — page didn't load (do NOT penalize — could be network/proxy)
arxiv_check: "no_id" — no arXiv ID to check

A mismatch does NOT by itself flip REAL→HALLUCINATION, but it is a serious citation error — surface it.

Check 1c — Venue consistency (always on; advisory / weak):

From the Check 1 search results, only if the venue or publisher is clearly shown and clearly conflicts with the cited venue, set venue_match: "no" and record what you saw in venue_found. Set "yes" only when results explicitly confirm the same venue; otherwise "unknown".

Do NOT flag preprint-vs-published or arXiv-year-vs-publication-year differences — those are normal and cause false alarms.
Only flag genuine conflicts (e.g., cited as CVPR but the paper is actually a NeurIPS paper).
venue_match NEVER changes the existence verdict — it is an advisory flag only.

Check 1d — Evidence URL reachability (ONLY if url_check_enabled; only for verdict: "real"):

WebFetch the recorded evidence URL and confirm it resolves and the page is about the cited paper.

evidence_check: "reachable_match" — resolves and content matches
evidence_check: "reachable_mismatch" — resolves but is a different paper → 🔴 surface it
evidence_check: "unreachable" — URL did not load
If the toggle is OFF, set evidence_check: "not_checked" (or omit the field).

Check 2 — Topic Relevance: Read the search result's abstract/summary. Compare the cited paper's actual topic to the citing paper's research domain. Do NOT assume relevance just because the paper exists.

Judge:

🎯 directly_relevant: Same research problem area; natural citation
🟡 somewhat_relevant: Related techniques or shared context; reasonable but not core
🔵 tangential: Same broad field but different specific problem; may be padding or background
🔴 unrelated: Completely different topic; possible citation error

Particular attention needed for:

Ethics/societal impact papers — often tangential, cited as background motivation
Method papers from other domains — may share a technique name but be about different problems
Non-academic references (blogs, GitHub wikis, model cards) — mark as tangential with low confidence
Preprints not yet peer-reviewed — note this in reasoning

2.3 Accumulate results

After each reference (incrementally), write/update $OUTDIR/{name}_results.json:

{
  "file": "paper.pdf",
  "extraction": {"total_references": N},
  "validations": [
    {"index": 1, "title": "...", "authors": [...], "date": "...",
     "venue": "...", "arxiv_id": "...",
     "verdict": "real", "confidence": 0.95,
     "reasoning": "Found on CVF with matching title and authors.",
     "evidence": ["https://..."],
     "arxiv_check": "match",
     "venue_match": "yes", "venue_found": "CVPR 2024",
     "evidence_check": "not_checked",
     "topic_relevance": "directly_relevant",
     "topic_relevance_score": 0.90,
     "topic_analysis": "Both papers address document dewarping...",
     "search_rounds": 1}
  ]
}

2.4 Generate report

python3 "$SKILL_DIR/tools/report_generator.py" \
    "$OUTDIR/{name}_results.json" \
    --output "$OUTDIR/{name}_VALIDATION_REPORT.md"

Key Rules

Validate EVERY reference individually — no batch-approving. 55 refs = 55 searches. If you skip any, say so explicitly in the report.
Check abstracts — a paper existing ≠ a paper being relevant. Read the search result summary to judge topic match.
Attribution errors = hallucination — real title + wrong authors is a hallucination.
Verify arXiv IDs — when a citation gives an arXiv ID, WebFetch arxiv.org/abs/{id} and confirm the title matches. An ID that resolves to a different paper (arxiv_check: "mismatch") is a serious citation error — surface it, even though the verdict can stay REAL.
Venue check is advisory only — flag venue_match: "no" only on a clear conflict; never flag preprint-vs-published or year differences, and never let it change the existence verdict.
Mark date discrepancies — note them but still mark as real if title+authors match.
Non-academic refs are flagged — blogs, GitHub wikis, model cards, tutorials → mark as tangential with appropriate confidence.

Scripts

| Script | Purpose | Dependencies | |--------|---------|-------------| | run_pipeline.py | Phase 1: PDF text extraction | pymupdf | | tools/report_generator.py | Phase 2: Generate MD report | none |

Academic Reference Validation

Recommended for