Documentation Audit Skill

Purpose

Audit and repair markdown documentation in codebases to ensure documentation claims match actual code behavior while maintaining documentation structure and completeness.

Success Criteria

1. Accuracy: All documentation claims verified against code
2. Navigation: Clear structure with functioning cross-links and README files where needed
3. Currency: Obsolete documentation archived or removed
4. Completeness: All significant features documented

Scope

Included:

• Markdown documentation files (*.md)
• Text-based diagrams (Mermaid, PlantUML, Graphviz, ASCII)
• Documentation structure and navigation

Excluded:

• Code comments and language-specific doc comments (godoc, JSDoc, etc.)
• Image-based diagrams
• Point-in-time documents (docs/plans/*)

When to Use This Skill

Use this skill when:

• Auditing documentation accuracy after significant code changes
• Onboarding to a new codebase and discovering documentation gaps
• Preparing for releases to ensure documentation is current
• Maintaining documentation as part of regular code health practices

Architecture: Multi-Pass Progressive Refinement

The skill operates in four passes, each building on the previous.

Pass 0: Repository Indexing

Build a comprehensive index of the repository before verification begins.

Create TodoWrite tasks for:

• Extract symbol graph using symbolic code analysis tools
• Identify API contracts from canonical sources (OpenAPI, protobuf, GraphQL schemas, configuration files)
• Map documentation cross-links and navigation files
• Index ownership metadata and last modification dates

Activities:

1. Use mcp__serena__find_symbol to extract exported symbols from each package
2. Use Glob to find canonical sources: **/*.proto, **/openapi.yaml, **/*.graphql, config schemas
3. Use Grep to map documentation links (search for [.*](.*\.md) patterns)
4. Note: Keep this pass lightweight - index only what's needed for verification

Output: Notes on repository structure, canonical sources found, and documentation topology.

Pass 1: Discovery and Classification

Discover all markdown files and classify them by purpose and lifecycle.

Create TodoWrite tasks for:

• Find all markdown files recursively
• Classify documents by type and lifecycle
• Identify obsolete candidates for archival

Activities:

1. Use Glob with pattern **/*.md to find all documentation
2. Classify using multiple signals:
   • Path heuristics: docs/archive/*, files with dates in names
   • Commit recency: Use Bash with git log --format=%ci --max-count=1 -- <file> to check last update
   • Link graph centrality: Documents frequently linked from others are likely living docs
   • YAML frontmatter: Look for status: living|archival|planned
3. Apply decision rules:
   • docs/plans/* are point-in-time (never updated)
   • High recency + clear structure = living documentation
   • Archived paths + low recency = obsolete candidate

Output: Classified document inventory with assessment of which docs to audit vs archive.

Safety: Move obsolete documents to docs/archive/ rather than delete. Ask user before archiving documents with ambiguous classifications.

Pass 2: Claim Extraction and Investigation Planning

Extract verifiable claims from living documentation and plan verification strategy.

Create TodoWrite tasks for each document section being validated.

Activities:

1. For each living documentation file:
   • Use Read to load the document
   • Extract claims by type:
     • Behavioral: "Service retries three times"
     • Structural: "Class implements Interface"
     • API: "Endpoint returns JSON with schema X"
     • Configuration: "Setting defaults to value Y"
     • Usage: "Run command with flag --foo"
2. Record claim metadata: doc_path, line_range, type, referenced_symbols
3. Identify text-based diagrams and classify:
   • Normative: Must match code structure exactly (UML class diagrams)
   • Illustrative: Conceptual, verify broad relationships only
4. Analyze codebase structure for documentation gaps:
   • Packages/modules without documentation
   • Exported APIs without usage examples
   • Directories lacking README files
5. Build investigation plan grouping claims by symbol/module for batch efficiency

Output: List of claims to verify, organized by module/package for efficient verification.

Optimization: Group related claims together to share context during verification.

Pass 3: Verification and Investigation

Verify claims against code using risk-appropriate methods.

Create TodoWrite tasks for each investigation batch.

Verification Hierarchy (use highest confidence method available):

1. Canonical sources: Verify against OpenAPI specs, protobuf definitions, GraphQL schemas, configuration schemas
2. Symbolic analysis: Verify structural claims using mcp__serena__find_symbol, mcp__serena__find_referencing_symbols, mcp__serena__search_for_pattern
3. Deep investigation: Use mcp__zen__analyze, mcp__zen__debug, or mcp__zen__thinkdeep for complex behavioral claims

Activities:

1. Process investigation batches from Pass 2
2. For each claim:
   • Select verification method based on claim type and available evidence
   • Start with symbolic analysis to narrow scope before deep investigation
   • Document findings with evidence trails (file paths, line numbers, symbol names)
   • Record whether claim is verified, contradicted, or requires user review
3. Verify diagrams:
   • Parse diagram syntax to extract assertions about code structure
   • Compare against symbol graph (normative) or verify broad structure (illustrative)
4. Create documentation for identified gaps:
   • Missing README files in documented directories
   • Undocumented features found in Pass 0 indexing
5. Handle conflicts:
   • Apply authority hierarchy: canonical sources > generated docs > top-level README > service READMEs
   • Record conflicts with context (version, environment, feature flags)
   • Annotate rather than delete when conflicts may be contextual

Output: Verified claims with corrections noted, new documentation drafts for gaps, conflict records.

Risk Management:

• Assign confidence to all verifications (exploring, low, medium, high, very_high, almost_certain, certain)
• Flag low-confidence verifications for user review
• Prefer static verification over runtime checks (safer, more deterministic)
• Never auto-fix based on weak evidence

Pass 4: Risk-Tiered Repair and Reporting

Apply corrections based on risk assessment.

Risk Tiers:

Auto-fix (apply immediately):

• Broken internal links (update paths)
• Typos in code references (symbol renamed in codebase)
• Outdated paths (files moved)
• Missing table of contents
• Diagram syntax errors

User approval required (present for review):

• Substantive technical corrections (behavior claims)
• Claim deletions or rewrites
• Document reclassifications
• Structural changes
• Conflict resolutions

User review required (flag but don't auto-fix):

• Low-confidence verifications
• Conflicting claims across documents
• Planned features (add disclaimers, don't verify behavior)
• Destructive actions (archival, removal)

Activities:

1. Apply auto-fixes immediately using Edit or Write tools
2. For user-approval changes:
   • Present diff preview
   • Show evidence trails (code spans, symbols verified, canonical sources)
   • Include confidence scores and rationale
   • Wait for user approval before applying
3. Move obsolete documents to docs/archive/ (after approval)
4. Create new README files for undocumented directories
5. Insert gap documentation using Write or Edit
6. Update navigation indexes
7. Generate summary report:
   • Changes by category and risk tier
   • Coverage metrics (claims verified / total claims)
   • Remaining manual review items
   • Confidence distribution

Output: Applied fixes, pending changes for review, archived documents, comprehensive summary.

Authority Hierarchy

When multiple documents make conflicting claims:

1. Canonical sources (OpenAPI, protobuf, schemas)
2. Generated documentation (godoc, JSDoc)
3. Top-level README.md
4. Service/module READMEs
5. Design documents
6. Ad-hoc notes

Document Classification Heuristics

Living Documentation (keep current):

• High commit frequency (updated within last 6 months)
• Clear ownership or maintenance signals
• Linked from README or navigation
• Path patterns: README.md, CONTRIBUTING.md, ARCHITECTURE.md, top-level docs

Point-in-Time (never update):

• Under docs/plans/
• Contains specific dates or version numbers
• Marked with status: plan or status: archival
• Low change frequency (<2 updates ever)

Obsolete (archive):

• No updates in 12+ months
• References removed code
• Superseded by newer documentation
• Marked with DRAFT or TODO

Skill Parameters

Accept these parameters when invoked:

• dry_run (default: true): Generate report without applying changes
• risk_tier (default: user_approval): Maximum risk tier to auto-apply (auto_fix | user_approval)
• verification_method (default: auto): Force specific verification approach (auto | canonical | symbolic | deep)
• focus_paths (default: all): Restrict to specific directories (e.g., "docs/", "README.md")
• skip_gap_analysis (default: false): Skip documentation gap detection

Usage Workflow

Step 1: Announce and Initialize

"I'm using the documentation-audit skill to verify documentation accuracy."

Create TodoWrite tasks for all four passes plus a completion task.

Step 2: Execute Pass 0 - Repository Indexing

Follow Pass 0 activities above. Keep this lightweight - just enough to understand repository structure and find canonical sources.

Step 3: Execute Pass 1 - Discovery and Classification

Follow Pass 1 activities above. Classify all markdown files and identify obsolete candidates.

Step 4: Execute Pass 2 - Claim Extraction

Follow Pass 2 activities above. Extract all verifiable claims and create investigation plan.

Step 5: Execute Pass 3 - Verification

Follow Pass 3 activities above. Verify each claim using appropriate verification hierarchy.

Step 6: Execute Pass 4 - Repair

Follow Pass 4 activities above. Apply fixes according to risk tiers and present changes for review.

Step 7: Summary

Provide conversational summary:

• Total claims verified
• Auto-fixes applied
• Changes requiring approval
• Items requiring manual review
• Overall documentation health assessment

Tools Available

• Serena symbolic tools: mcp__serena__find_symbol, mcp__serena__find_referencing_symbols, mcp__serena__search_for_pattern, mcp__serena__get_symbols_overview
• Zen investigation tools: mcp__zen__analyze, mcp__zen__debug, mcp__zen__thinkdeep, mcp__zen__codereview
• File operations: Read, Write, Edit, Glob, Grep
• Version control: Bash with git commands for commit history

Token Efficiency Strategies

1. Index once, reuse: Pass 0 creates repository knowledge for later passes
2. Batch similar claims: Group by symbol/module to share context
3. Retrieval-augmented: Fetch only relevant code spans for verification
4. Prefer static: Use symbolic analysis before deep investigation
5. Focused reading: Use mcp__serena__get_symbols_overview before reading full files

Safety Mechanisms

1. Evidence trails: Every change documents source claim, verification method, evidence
2. Confidence scoring: Low confidence blocks auto-fixes
3. Diff previews: Show changes before applying
4. Archival over deletion: Move obsolete docs, don't delete
5. Risk tiers: Separate auto-fixable issues from those requiring approval

Edge Cases

Feature flags: Claims may be contextual (true when flag enabled). Record context, don't mark as contradicted.

Multiple versions: Documentation for different versions may coexist. Partition claims by version.

Planned features: Add disclaimer banners, mark verification: not_applicable. Don't attempt behavioral verification.

External dependencies: Claims about external APIs may be stale. Verify against canonical sources if available, otherwise flag for review.

Runtime behavior: Claims requiring execution (performance, flakiness) are fragile. Prefer symbolic verification or mark for manual testing.

Example Execution

User: Run documentation audit on docs/ with auto-fixes enabled

Claude:
1. "I'm using the documentation-audit skill to verify documentation accuracy."
2. Creates TodoWrite for 4 passes + summary
3. Pass 0: Indexes repository (symbols, canonical sources, doc links)
4. Pass 1: Finds 23 markdown files, classifies 3 as obsolete candidates
5. Pass 2: Extracts 156 claims from 20 living docs, groups into 12 investigation batches
6. Pass 3: Verifies claims using symbolic analysis (78%), canonical sources (12%), deep investigation (10%)
   - Finds 8 contradicted claims, 12 documentation gaps
7. Pass 4:
   - Auto-fixes: 3 broken links, 1 outdated path
   - Presents for approval: 8 technical corrections, 12 new documentation sections
   - Flags for review: 2 low-confidence verifications
8. Summary: "Verified 156 claims with 95% confidence. Applied 4 auto-fixes, 20 changes await approval, 2 require manual review."

Anti-Patterns

❌ Don't skip symbolic analysis - Use serena tools before reading full files ❌ Don't auto-fix low-confidence findings - Flag for user review instead ❌ Don't delete documents - Archive to docs/archive/ instead ❌ Don't batch all changes - Apply auto-fixes immediately, present substantive changes for review ❌ Don't verify point-in-time documents - Leave docs/plans/* alone ❌ Don't skip TodoWrite - Track progress through all four passes

Summary

This skill performs rigorous documentation audits through four systematic passes:

1. Index repository structure and canonical sources
2. Classify documents by lifecycle and purpose
3. Extract and group verifiable claims
4. Verify claims and apply risk-tiered repairs

90% of effort is verification against code, 10% is structure and quality.