CiteCheck — Paper Citation Verification

Overview

CiteCheck verifies academic paper citations by combining structured parsing with agent-native LLM evaluation. It supports LaTeX source files (preferred) and PDF fallback.

Workflow

1. Parse paper: Call citecheck CLI to read LaTeX (preferred) or PDF, extract references and body text
2. Format check: Call citecheck CLI to validate bibliography entries
3. Queryability verification: Call citecheck CLI to verify existence via Crossref / Semantic Scholar
4. Evaluate thematic relevance: Evaluate directly using the agent's reasoning capabilities — compare cited paper title/abstract/venue against the citing paper
5. Evaluate semantic accuracy: Evaluate directly using the agent's reasoning capabilities — compare in-text citation context against cited source content
6. Generate report: Aggregate all results into a Markdown report

Why matching steps are not done by the CLI The citecheck CLI can run standalone with optional --api-key for external LLM-powered matching. When used as a Skill, the host agent itself possesses LLM reasoning capabilities. Direct evaluation is faster, more consistent, and requires no additional API keys from the user.

---

1. Parse Paper (CLI)

LaTeX source (preferred)

citecheck path/to/latex_project/ --skip-verification --skip-semantic -o parsed_report.md

Or parse a single file:

citecheck main.tex --skip-verification --skip-semantic -o parsed_report.md

PDF (fallback)

citecheck paper.pdf --skip-verification --skip-semantic -o parsed_report.md

--skip-verification and --skip-semantic are required in Skill mode because steps 3–5 are performed directly by the agent.

---

2. Format Check (CLI)

Call citecheck to check and report format issues:

Check item	Description
Required fields	Author, title, year, venue completeness
Format consistency	Punctuation, capitalization, abbreviation uniformity
DOI/URL	If present, whether format is correct and accessible
Year sanity	No 202x placeholders, not in the future

Detailed rules: see ./references/format-check-rules.md.

---

3. Queryability Verification (CLI + Agent Supplement)

Call citecheck to query Crossref and Semantic Scholar public APIs for citation existence.

API details: see ./references/api-reference.md.

If the CLI fails due to network/SSL issues, use WebSearch to directly query suspicious citations (especially those marked "unverifiable"), supplementing the verification results.

---

4. Evaluate Thematic Relevance (Agent Direct)

Do not call the CLI or any external API. Evaluate directly using the agent's reasoning capabilities.

For each reference, extract:

• Citing paper: title, abstract, keywords
• Cited paper: title, abstract (from API results or WebSearch), venue

Evaluation prompt template: see ./references/thematic-scoring-prompt.md.

---

5. Evaluate Semantic Accuracy (Agent Direct)

Do not call the CLI or any external API. Evaluate directly using the agent's reasoning capabilities.

For each in-text citation position:

1. Extract 1–2 sentences before and after the citation marker as citing_text
2. Obtain the cited paper's abstract via Semantic Scholar or WebSearch as cited_text
3. Evaluate semantic consistency using the prompt template: see ./references/semantic-matching-prompt.md.

---

6. Output Report

Generate a Markdown report containing:

Summary

• Total references
• Format issues count
• Query failures / suspicious count
• Average thematic relevance score
• Average semantic accuracy score

Detailed Results Table

No.	Title	Format	Queryable	Thematic	Semantic	Notes
1	...	✅/⚠️/❌	✅/❌	0.85	0.90	DOI mismatch

Issue Summary

• List all findings and recommendations by severity

---

Dependencies

pip install CiteCheck

For PDF support:

pip install CiteCheck[pdf]

---

Notes

• No --api-key is required in Skill mode. Thematic and semantic matching are performed directly by the agent, with no external LLM API calls.
• Prioritize LaTeX source files; parsing accuracy is far higher than PDF.
• Add delays and retries to API calls to avoid rate limiting.
• When Semantic Scholar abstracts are missing, fall back to Crossref + title keywords.
• Semantic matching requires the cited paper's abstract; mark "source unreachable" when unavailable.
• If placeholder citations like =? or [?, ?] are found, mark them as "unverifiable references".