Dismech Reference Validation Skill

Overview

Validate and repair evidence references in the dismech disorder knowledge base. This ensures that quoted snippets actually appear in the cited PubMed abstracts, preventing fabricated or misquoted evidence from entering the knowledge base.

When to Use

• Validating evidence items after adding new disorder content
• Checking that snippets match their cited PMID abstracts
• Repairing evidence items with minor text mismatches
• Removing fabricated evidence (AI hallucinations)
• QC checks before committing changes

Evidence Item Structure

All evidence items follow this YAML structure:

evidence:
  - reference: PMID:12345678
    supports: SUPPORT  # SUPPORT, REFUTE, PARTIAL, NO_EVIDENCE, WRONG_STATEMENT
    snippet: "Exact quoted text from the abstract"
    explanation: "Why this evidence supports/refutes the claim"

Support Classifications

Validation Commands

Validate a Single File

uv run linkml-reference-validator validate data kb/disorders/Asthma.yaml \
  --schema src/dismech/schema/dismech.yaml \
  --target-class Disease

Validate All Disorder Files

just validate-all

Or manually:

for f in kb/disorders/*.yaml; do
  echo "=== $f ==="
  uv run linkml-reference-validator validate data "$f" \
    --schema src/dismech/schema/dismech.yaml \
    --target-class Disease
done

Using the Just Target

just qc  # Runs all QC including reference validation

Repair Commands

Dry Run (Preview Changes)

uv run linkml-reference-validator repair data kb/disorders/Cholera.yaml \
  --schema src/dismech/schema/dismech.yaml \
  --target-class Disease

Auto-Repair with Threshold

uv run linkml-reference-validator repair data kb/disorders/Cholera.yaml \
  --schema src/dismech/schema/dismech.yaml \
  --target-class Disease \
  --no-dry-run \
  --fix-threshold 0.80

The --fix-threshold 0.80 means snippets with 80%+ similarity to the actual abstract text will be automatically corrected.

Common Error Patterns

1. Snippet Not Found in Abstract

ERROR: Snippet not found in reference PMID:12345678
  Snippet: "The patient showed symptoms..."
  Abstract: [actual abstract text]

Solutions:

• Check if snippet is from full text (not abstract) - may need to remove
• Check for minor typos - use repair with threshold
• If fabricated, remove the evidence item entirely

2. Reference Cannot Be Fetched

ERROR: Could not fetch reference PMID:99999999

Solutions:

• Verify PMID exists on PubMed
• Check for typos in PMID
• If PMID is invalid, remove the evidence item

3. Fabricated Evidence Patterns

Watch for these red flags indicating AI-generated fake evidence:

• Snippet says "N/A" or "No abstract available"
• Snippet is suspiciously perfect match to the claim
• PMID doesn't exist or is for unrelated topic
• Generic statements without specific data

Solution: Remove the entire evidence item.

Cache Management

Reference validator caches PubMed abstracts in .refval_cache/. If you encounter stale cache issues:

rm -rf .refval_cache/

Cache File Format Issues

If you see YAML parsing errors in cache files, check for unquoted colons in titles:

# Bad - will cause parse error
title: COVID-19: A New Challenge

# Good - properly quoted
title: "COVID-19: A New Challenge"

Batch Processing Workflow

1. Get Error Count

uv run linkml-reference-validator validate data kb/disorders/*.yaml \
  --schema src/dismech/schema/dismech.yaml \
  --target-class Disease 2>&1 | grep -c "ERROR"

2. Process Files with Errors

for f in kb/disorders/*.yaml; do
  errors=$(uv run linkml-reference-validator validate data "$f" \
    --schema src/dismech/schema/dismech.yaml \
    --target-class Disease 2>&1 | grep -c "ERROR" || echo 0)
  if [ "$errors" -gt 0 ]; then
    echo "=== $f has $errors errors ==="
  fi
done

3. Auto-Repair All

for f in kb/disorders/*.yaml; do
  uv run linkml-reference-validator repair data "$f" \
    --schema src/dismech/schema/dismech.yaml \
    --target-class Disease \
    --no-dry-run \
    --fix-threshold 0.80
done

Best Practices

Adding New Evidence

1. Use real PMIDs: Always verify the PMID exists on PubMed
2. Quote exactly: Copy snippet text directly from the abstract
3. Keep snippets short: 1-2 sentences that directly support the claim
4. Validate immediately: Run validation after adding evidence

Reviewing AI-Generated Content

When reviewing disorder files that may contain AI-generated evidence:

1. Run validation first to catch obvious fabrications
2. Spot-check PMIDs on PubMed
3. Look for suspiciously perfect or generic snippets
4. Remove any evidence that cannot be verified

Handling Unfetchable References

If a reference cannot be fetched:

1. Manually check PubMed for the PMID
2. If it exists but is restricted, note in explanation
3. If it doesn't exist, remove the evidence item
4. Consider replacing with a valid alternative reference

Integration with Schema

The evidence structure is defined in src/dismech/schema/dismech.yaml:

EvidenceItem:
  attributes:
    reference:
      description: PMID or DOI reference
      pattern: "^PMID:\\d+$|^DOI:.*$"
    supports:
      range: SupportType
    snippet:
      description: Quoted text from the reference
    explanation:
      description: Why this evidence supports/refutes the claim