Claude Code Plugins

Community-maintained marketplace

Feedback

Eval Gap Finder

@sunholo-data/ailang
13
0

Find AILANG vs Python eval gaps and improve prompts/language. Use when user says 'find eval gaps', 'analyze benchmark failures', 'close Python-AILANG gap', or after running evals.

Install Skill

1Download skill
2Enable skills in Claude

Open claude.ai/settings/capabilities and find the "Skills" section

3Upload to Claude

Click "Upload skill" and select the downloaded ZIP file

Note: Please verify skill by going through its instructions before using it.

SKILL.md

name Eval Gap Finder
description Find AILANG vs Python eval gaps and improve prompts/language. Use when user says 'find eval gaps', 'analyze benchmark failures', 'close Python-AILANG gap', or after running evals.

Eval Gap Finder

Automates the process of finding and closing the gap between Python and AILANG benchmark success rates. Identifies language limitations, prompt gaps, and missing stdlib functions.

Quick Start

Most common usage:

# User says: "Find eval gaps" or "Analyze benchmark failures"
# This skill will:
# 1. Run evals with dev models (gemini-3-flash, claude-haiku-4-5)
# 2. Compare Python vs AILANG success rates
# 3. Identify benchmarks where Python passes but AILANG fails
# 4. Analyze error patterns and categorize them
# 5. Check if gaps are documented in prompt
# 6. Test proposed examples and add to prompt
# 7. Create design docs for language limitations

When to Use This Skill

Invoke this skill when:

  • User asks to "find eval gaps" or "close the Python-AILANG gap"
  • User wants to analyze benchmark failures
  • After running evals and seeing lower AILANG success
  • User says "why is AILANG failing?" or "improve AILANG benchmarks"
  • User wants to identify language limitations

Available Scripts

scripts/run_gap_analysis.sh [eval_dir]

Run full gap analysis on eval results.

.claude/skills/eval-gap-finder/scripts/run_gap_analysis.sh eval_results/v0.6.5

scripts/identify_python_only.sh <eval_dir>

List benchmarks where Python passes but AILANG fails.

.claude/skills/eval-gap-finder/scripts/identify_python_only.sh eval_results/v0.6.5

scripts/categorize_errors.sh <eval_dir>

Categorize AILANG failures by error type.

.claude/skills/eval-gap-finder/scripts/categorize_errors.sh eval_results/v0.6.5

scripts/test_example.sh <code>

Test if an AILANG code example compiles and runs correctly.

.claude/skills/eval-gap-finder/scripts/test_example.sh /tmp/test.ail

Workflow

1. Run Evals with Dev Models

ailang eval-suite --models gemini-3-flash,claude-haiku-4-5 --output eval_results/gap-analysis

Target: Run with cheap/fast models first. If they succeed, larger models should too.

2. Generate Summary and Identify Gaps

ailang eval-summary eval_results/gap-analysis
.claude/skills/eval-gap-finder/scripts/identify_python_only.sh eval_results/gap-analysis

Key metrics:

  • AILANG success rate (target: >70%)
  • Python success rate (baseline)
  • Gap (python_only benchmarks)

3. Analyze Error Patterns

For each Python-only pass, categorize the error:

Category Pattern Fix Approach
WRONG_LANG Model wrote Python syntax Stronger "NOT Python" in prompt
PAR_001 Parse errors (syntax) Add more examples to prompt
Type errors Type unification failures May be language limitation
Logic errors Compiles but wrong output Better examples or algorithm
EOF errors Incomplete code generation Model limitation, not prompt

4. Check Prompt Coverage

For each gap, check if the pattern is documented:

grep -n "pattern" prompts/v0.6.5.md

If not documented, add:

  • Working example to Quick Reference section
  • Entry in "What AILANG Does NOT Have" table (if limitation)
  • New section if pattern is complex

5. Test Examples Before Adding

CRITICAL: Always test examples before adding to prompt!

cat > /tmp/test.ail << 'EOF'
module benchmark/solution
-- Your example code here
EOF
ailang run --caps IO --entry main /tmp/test.ail

If example fails, it reveals a language gap - create a design doc instead.

6. Create Design Docs for Language Gaps

If testing reveals a language limitation:

  1. Create design doc: design_docs/planned/vX_Y_Z/m-<feature>.md
  2. Document:
    • Minimal reproduction
    • Error message
    • Workaround (for prompt)
    • Proposed fix
  3. Add workaround to prompt with note

7. Track Improvement

After updates, re-run evals:

ailang eval-suite --models gemini-3-flash,claude-haiku-4-5 --output eval_results/gap-analysis-v2

Compare:

  • Success rate improvement
  • Which benchmarks fixed
  • Any regressions

Error Categories Reference

Error Meaning Fix
WRONG_LANG Wrote Python instead Prompt emphasis
PAR_001 Parser error Syntax examples
PAR_UNEXPECTED_TOKEN Wrong token Syntax examples
TC_* Type check error Type examples or design doc
"undefined variable" Missing import/letrec Document pattern
EOF errors Incomplete code Model limitation
logic_error Wrong output Algorithm examples

Resources

Gap Analysis Template

See `resources/gap_analysis_template.md` for structured analysis format.

Common Patterns

See `resources/common_patterns.md` for frequently encountered gaps.

Progressive Disclosure

This skill loads information progressively:

  1. Always loaded: This SKILL.md file (workflow overview)
  2. Execute as needed: Scripts in scripts/ directory
  3. Load on demand: Resources for templates and patterns

Notes

  • Always test examples before adding to prompt
  • Prefer fixing language over prompt workarounds
  • Track improvements with before/after eval runs
  • Create design docs for language limitations
  • Update prompt hash in versions.json after changes