Base Model Selector + Baseline Evaluation

Overview

Evaluate base models against your domain rubric BEFORE committing to fine-tuning. Most projects skip this and waste resources fine-tuning models that were already good enough—or fine-tuning models that need fundamental architectural changes instead.

Core principle: Never fine-tune without baseline data. The baseline tells you if fine-tuning will help and by how much.

When to Use

digraph when_to_use {
    "Starting fine-tuning project?" [shape=diamond];
    "Have baseline metrics?" [shape=diamond];
    "Use this skill" [shape=box, style=filled, fillcolor=lightgreen];
    "Skip - you have data" [shape=box];
    "Proceed to data generation" [shape=box];

    "Starting fine-tuning project?" -> "Have baseline metrics?" [label="yes"];
    "Starting fine-tuning project?" -> "Skip - you have data" [label="no"];
    "Have baseline metrics?" -> "Proceed to data generation" [label="yes"];
    "Have baseline metrics?" -> "Use this skill" [label="no"];
}

Use when:

• Starting any fine-tuning project
• Switching base models mid-project
• Evaluating if existing model needs fine-tuning for new domain

Skip when:

• You already have baseline evaluation data
• Pure prompting project (no fine-tuning planned)

The Process

Step 1: Deep Research (EXHAUSTIVE)

This is NOT quick research. You must build a comprehensive comparison table covering ALL viable candidates for your target size and use case.

1a. Identify ALL Candidates

Search for models in your target parameter range. For 7-10B, check:

Search queries to run:

• "best [size] parameter LLM 2025 [your domain]"
• "[model family] vs [model family] comparison 2025"
• "open source LLM [your domain] fine-tuned 2025"
• "best LLM for [specific capability] 2025" (e.g., empathy, coding, reasoning)

1b. Evaluate Each Factor

1c. Domain-Specific Research

CRITICAL: Different domains need different model characteristics.

Example research for therapeutic coaching:

"Models such as DeepSeek R1, OpenAI o3, and o1 excel in logical reasoning but fall short in conversational fluency and emotional responsiveness."

This means: avoid DeepSeek R1 distills for therapy, even though they're excellent models.

1d. Build Comparison Table

Create a comprehensive table with ALL candidates:

| Model | Params | License | Context | Release | Domain Fit | Notes |
|-------|--------|---------|---------|---------|------------|-------|
| Qwen 3 8B | 8.2B | Apache 2.0 | 128K | Apr 2025 | ⭐⭐⭐⭐ | Thinking/non-thinking modes |
| ... | ... | ... | ... | ... | ... | ... |

Rate domain fit on a scale (⭐ to ⭐⭐⭐⭐⭐) based on research findings.

Step 2: Select Primary Candidate

Based on research, select ONE candidate. Document:

• Why this model over alternatives
• Expected strengths for your domain
• Potential concerns to watch for

Also identify a backup candidate in case primary underperforms.

Step 3: Pull Model

# Ollama
ollama pull qwen3:8b

# Or llama.cpp with GGUF
huggingface-cli download Qwen/Qwen3-8B-Instruct-GGUF qwen3-8b-instruct-q4_k_m.gguf

# Verify it runs
llama-cli -m model.gguf -p "Hello, testing."

Step 4: Generate Evaluation Scenarios

Create ~50 diverse scenarios from your input taxonomy. These are INPUTS only (user messages), not full conversations.

# Generate opening messages covering your taxonomy
scenarios = []
for topic in taxonomy["topics"]:
    for subtopic in topic["subtopics"]:
        for style in ["terse", "conversational", "detailed"]:
            scenarios.append(generate_opening(topic, subtopic, style))

Critical: Scenarios must cover your full input distribution, including edge cases.

Step 5: Run Base Model

Run the base model on each scenario. Collect single-turn responses.

responses = []
for scenario in scenarios:
    response = generate(
        model=model_path,
        prompt=scenario,
        system=your_system_prompt,
    )
    responses.append({
        "scenario": scenario,
        "response": response,
    })

Step 6: Assess with Rubric

Run your domain rubric on each response. Calculate pass rate.

results = []
for item in responses:
    assessment = assess_single_turn(item["scenario"], item["response"])
    results.append(assessment)

pass_rate = sum(1 for r in results if r.passed) / len(results)

Step 7: Make Decision

digraph decision {
    "pass_rate" [shape=diamond, label="Pass Rate?"];
    "qualitative" [shape=box, label="≥70%: Qualitative review\nMay not need fine-tuning"];
    "proceed" [shape=box, label="50-70%: Proceed with\nfine-tuning (moderate gains)"];
    "full" [shape=box, label="<50%: Full pipeline\n(significant gains possible)"];

    "pass_rate" -> "qualitative" [label="≥70%"];
    "pass_rate" -> "proceed" [label="50-70%"];
    "pass_rate" -> "full" [label="<50%"];
}

Step 8: Document Results

Create docs/base-model-evaluation.md:

# Base Model Evaluation

**Date:** YYYY-MM-DD

## Model Selection

### Research Summary

[Summary of models considered and why primary was chosen]

| Model | Domain Fit | Why Considered | Why Selected/Rejected |
|-------|------------|----------------|----------------------|
| Qwen 3 8B | ⭐⭐⭐⭐ | Latest gen, non-thinking mode | **SELECTED** - best chat quality |
| DeepSeek-R1-7B | ⭐⭐ | Strong reasoning | Rejected - poor empathy per research |
| ... | ... | ... | ... |

### Selected Model

- **Model:** qwen3:8b-instruct-q4_k_m
- **Parameters:** 8.2B
- **License:** Apache 2.0
- **Context:** 128K tokens

## Evaluation Results

- **Scenarios:** 50
- **Pass rate:** XX%
- **Decision:** [DEPLOY / FINE-TUNE / FULL PIPELINE]

## Failure Analysis

| Criterion | Failure Count | Pattern |
|-----------|---------------|---------|
| CQ3 | 12 | Jumps to advice without validation |
| CQ8 | 3 | Missed crisis signals |

## Qualitative Notes

[Specific observations about response quality]

## Next Steps

[What happens next based on decision]

Outputs

1. Research table — All candidates with domain fit ratings
2. Selected model — Name, version, and rationale
3. Baseline pass rate — Percentage on rubric
4. Failure analysis — Which criteria fail most
5. Decision document — docs/base-model-evaluation.md

Common Mistakes

Quick Reference

# 1. Research (use web search extensively)
# Build comparison table with 8-12 candidates

# 2. Pull model
ollama pull qwen3:8b
# or
huggingface-cli download Qwen/Qwen3-8B-Instruct-GGUF

# 3. Generate scenarios
uv run python generator.py --scenarios-only 50

# 4. Run evaluation
uv run python evaluate_base_model.py --model qwen3:8b

# 5. Check results
cat docs/base-model-evaluation.md