Educational Demo Evaluator

Execute test cases. Verify learning outcomes. Score quality vs benchmark.

Core Principles

1. Execute test_cases.json - don't invent tests
2. Use Chrome tools - real browser interaction
3. Verify LEARNING - not just button clicks
4. Score vs benchmark - quality comparison

Screenshot System (REQUIRED)

Critical: Next generation builders study these screenshots. You MUST organize them:

1. During test execution, click "📸 Capture State" button at key moments
2. After testing, click "⬇️ Download Screenshots" button
3. Screenshots download to ~/Downloads/ with labels (capture_1.png, etc.)
4. MOVE them to /problems/<name>/screenshots/agent_X_*.png
5. Reference filenames in your evaluation JSON output

Why required:

• Gen 2 builders read screenshots/ to understand which vibes worked visually
• LESSONS_LEARNED tells them what worked; screenshots show them how
• Builders need both to discover their approach for the next generation

Example structure:

/problems/quicksort-demo/screenshots/
├── agent_1_initial.png (gen1, narrative vibe)
├── agent_1_step_1.png
├── agent_4_initial.png (gen1, comparison vibe)
├── agent_4_step_1.png
...

Prerequisites

Before evaluation:

1. test_cases.json must exist in problem folder
2. Demo HTML file must exist
3. Benchmark screenshots for quality reference

Workflow

Step 1: Setup Chrome

# Get or create tab
mcp__claude-in-chrome__tabs_context_mcp(createIfEmpty=true)
# Returns: tabId

# Create new tab for testing
mcp__claude-in-chrome__tabs_create_mcp()
# Returns: new tabId - use this one

Step 2: Load Test Cases

Read(problems/<name>/test_cases.json)

Step 3: Load Benchmark (quality reference)

Read(problems/<name>/benchmark_ux/*.png)
# Note the QUALITY level to compare against

Step 4: Navigate to Demo

mcp__claude-in-chrome__navigate(
  url="file:///absolute/path/to/agent.html",
  tabId=X
)

# Wait for load
mcp__claude-in-chrome__computer(
  action="wait",
  duration=2,
  tabId=X
)

# Screenshot initial state
mcp__claude-in-chrome__computer(
  action="screenshot",
  tabId=X
)

Step 5: Execute Each Test Case

For each test case in test_cases.json:

# Read page structure
mcp__claude-in-chrome__read_page(tabId=X, filter="interactive")

# Execute steps from test case
for step in test_case.steps:
  if step.action == "find":
    result = mcp__claude-in-chrome__find(query=step.query, tabId=X)

  elif step.action == "input":
    mcp__claude-in-chrome__form_input(ref=found_ref, value=step.value, tabId=X)

  elif step.action == "click":
    mcp__claude-in-chrome__computer(action="left_click", ref=found_ref, tabId=X)

  elif step.action == "wait":
    mcp__claude-in-chrome__computer(action="wait", duration=step.ms/1000, tabId=X)

  elif step.action == "screenshot":
    # Use the built-in Capture State button
    find_result = mcp__claude-in-chrome__find(
      query="Capture State button",
      tabId=X
    )
    mcp__claude-in-chrome__computer(
      action="left_click",
      ref=find_result['ref'],
      tabId=X
    )

Step 5b: Download Screenshots

After executing all test cases:

# Click Download Screenshots button
find_result = mcp__claude-in-chrome__find(
  query="Download Screenshots button",
  tabId=X
)
mcp__claude-in-chrome__computer(
  action="left_click",
  ref=find_result['ref'],
  tabId=X
)

# Screenshots automatically download with labels
# (initial_state.png, step_1.png, etc.)

Step 6: Organize Screenshots

After downloading, organize for next generation builders:

# Screenshots are in ~/Downloads/
# Move them to /problems/<name>/screenshots/ with agent labels

mv ~/Downloads/capture_1.png /problems/<name>/screenshots/agent_1_initial.png
mv ~/Downloads/capture_2.png /problems/<name>/screenshots/agent_1_step_1.png
...

Required naming: agent_{id}_{label}.png so builders can find them by agent.

Step 7: Verify Learning Outcomes

After organizing screenshots:

1. Compare against test_case.verify expectations
2. Check: Did the demo teach correctly?
3. Review both visual (screenshots) and learning outcomes

Questions to answer:
- Does visual match expected? (e.g., "5 bubbled up to root")
- Is the learning outcome achieved?
- Would a learner understand the concept?

Chrome Tools Reference

Example Evaluation Session

# Setup
tabs_context_mcp(createIfEmpty=true) -> existing tabs
tabs_create_mcp() -> tabId: 456

# Navigate
navigate(url="http://localhost:9999/problems/heap-demo/generations/gen1/agent_1.html", tabId=456)

# Wait and let demo initialize
computer(action="wait", duration=2, tabId=456)

# Read structure
read_page(tabId=456, filter="interactive")
-> ref_1: textbox "value input"
-> ref_2: button "Insert"
-> ref_3: button "Extract Min"
-> ref_4: button "📸 Capture State"

# Execute test case: insert_bubble_up
form_input(ref="ref_1", value="5", tabId=456)
computer(action="left_click", ref="ref_2", tabId=456)
computer(action="wait", duration=2, tabId=456)

# Capture at this key moment
computer(action="left_click", ref="ref_4", tabId=456)  # Click Capture State button
computer(action="wait", duration=1, tabId=456)

# Verify
# Check: Did 5 bubble up correctly? Is animation visible?

# ... test more cases ...

# When done, download screenshots
find_result = find(query="Download Screenshots button", tabId=456)
computer(action="left_click", ref=find_result['ref'], tabId=456)

# Then move from ~/Downloads to /problems/heap-demo/screenshots/agent_1_*.png

Viewport Verification

Check viewport issues during testing:

# Read full page
read_page(tabId=X, filter="all")

# Check for scroll indicators
# Check if elements are cut off
# Check if buttons are accessible

# Screenshot full page area
computer(action="screenshot", tabId=X)
# Verify: Is everything visible without scrolling?

Score Categories (out of 100)

Automatic Deductions

Output Format

{
  "agent": "gen2/agent_3.html",
  "benchmarks_read": ["08_heaps.png"],
  "test_cases_executed": [
    {
      "id": "insert_bubble_up",
      "result": "PASS",
      "screenshots": ["capture_1.png", "capture_2.png"],
      "screenshot_notes": "Initial state and after insert step captured in ~/Downloads/",
      "learning_verified": true
    },
    {
      "id": "extract_root",
      "result": "FAIL",
      "reason": "Animation skips comparison step",
      "screenshots": ["capture_3.png", "capture_4.png"],
      "screenshot_notes": "Before extract and after extract in ~/Downloads/",
      "learning_verified": false
    }
  ],
  "viewport_check": {
    "fits_viewport": true,
    "issues": []
  },
  "scores": {
    "correctness": 20,
    "clarity": 18,
    "educational_value": 12,
    "viewport": 15,
    "interaction": 13
  },
  "total": 78,
  "bugs": [],
  "correctness_issues": ["extract animation incomplete"]
}

Reality Check

• If ANY test case fails correctness: max score 50
• Most demos score 30-50
• 70+ requires ALL test cases passing
• Compare screenshots to benchmark for quality judgment

Cleanup

After evaluation:

# Close the test tab (optional)
# Or leave open for debugging