Model Manager

Test API access, validate configurations, and add new AI models to the AILANG eval suite.

Quick Start

Most common usage:

# User says: "Can we add GPT-5.1 to the eval suite?"
# This skill will:
# 1. Test API access to GPT-5.1
# 2. Find the correct API model name
# 3. Look up pricing information
# 4. Update models.yml configuration
# 5. Run a test benchmark to verify

When to Use This Skill

Invoke this skill when:

• User asks to "add a new model" to eval suite
• User mentions checking if a model is "accessible" or "available"
• User wants to "test API access" to a model
• User asks to "update models.yml" or "check pricing"
• User says "can we use [model name]?" for evaluations

Available Scripts

scripts/test_model_access.sh <provider> <model-name>

Test API access to a model and display authentication status.

Usage:

# Test OpenAI model
scripts/test_model_access.sh openai gpt-5.1

# Test Anthropic model
scripts/test_model_access.sh anthropic claude-sonnet-5-5

# Test Google Gemini via Vertex AI
scripts/test_model_access.sh google gemini-3-pro-preview-11-2025
# Test Ollama Cloud (flat-rate open-weight route; tag WITHOUT :cloud)
scripts/test_model_access.sh ollama-cloud glm-5.3-flash

Output:

Testing: openai/gpt-5.1
✓ OPENAI_API_KEY found
✓ API call successful
✓ Model: gpt-5.1-2025-11-13
✓ Tokens: 13 input, 10 output (10 reasoning)
Ready to add to models.yml

scripts/find_model_info.sh <model-keywords>

Search for model information using web search and return API names + pricing.

Usage:

# Find GPT-5.1 info
scripts/find_model_info.sh "GPT-5.1 API model name pricing"

# Find Gemini 3 Pro info
scripts/find_model_info.sh "Gemini 3 Pro API documentation"

Output:

Searching for: GPT-5.1 API model name pricing
✓ Found API names:
  - gpt-5.1 (Thinking mode)
  - gpt-5.1-chat-latest (Instant mode)
✓ Pricing:
  Input: $1.25 per 1M tokens
  Output: $10.00 per 1M tokens
  Cached: $0.125 per 1M tokens

scripts/update_models_yml.sh <friendly-name> <api-name> <provider> <input-price> <output-price>

Add a new model to models.yml configuration.

Usage:

# Add GPT-5.1
scripts/update_models_yml.sh \
  gpt5-1 \
  "gpt-5.1" \
  openai \
  0.00125 \
  0.01

Output:

Adding model to models.yml:
  Friendly name: gpt5-1
  API name: gpt-5.1
  Provider: openai
  Pricing: $0.00125 / $0.01 per 1K tokens

✓ Updated models.yml
✓ Validated YAML syntax
✓ Ready to test

scripts/verify_vertex_model.sh <model-name>

Check if a Gemini model is available in Vertex AI.

Usage:

# Check if Gemini 3 Pro is available
scripts/verify_vertex_model.sh gemini-3-pro-preview-11-2025

Output:

Checking Vertex AI for: gemini-3-pro-preview-11-2025
✓ GCP project: multivac-internal-prod
✓ Access token obtained
✗ Model not found (404)
Recommendation: Monitor for availability, check again in 1-2 weeks

scripts/measure_ollama_credit_rate.sh <tag1> [tag2 ...]

Ollama Cloud credit burn per token across models — see resources/ollama_cloud.md.

scripts/run_test_benchmark.sh <model-name>

Run a small test benchmark to verify model works end-to-end.

Usage:

# Test GPT-5.1 with fizzbuzz benchmark
scripts/run_test_benchmark.sh gpt5-1

Output:

Running test benchmark: fizzbuzz
Model: gpt5-1
✓ Benchmark completed
✓ Result: PASS (100%)
✓ Tokens: 245 input, 89 output
✓ Cost: $0.002
Model is ready for production use

Workflow

1. Test API Access

First, verify you can call the model:

# Use test_model_access.sh
scripts/test_model_access.sh openai gpt-5.1

Credentials live in ~/.config/ailang/secrets.env (ANTHROPIC/OPENAI/GOOGLE/OPENROUTER/ OLLAMA keys). An interactive shell does not source it — . ~/.config/ailang/secrets.env before eval-suite or the access scripts. Without ANTHROPIC_API_KEY, standard mode falls back to the Claude subscription OAuth token, which the provider serves for Haiku only: Sonnet 5.5 / Opus 5.5 come back 429 rate_limit_error "Error" with no rate-limit headers (measured 2026-09-29). A 429 with no anthropic-ratelimit-* headers is that, not quota.

What to check:

• API key is set (OPENAI_API_KEY, ANTHROPIC_API_KEY, or gcloud auth)
• API call succeeds (not 401/403/404)
• Model returns expected structure
• Token usage is reported

For Claude models — also probe the agent lane (claude CLI, subscription):

env -u ANTHROPIC_API_KEY claude -p --model <full-id> --output-format json "Reply OK" | jq '.modelUsage|keys'
env -u ANTHROPIC_API_KEY claude -p --model claude-sonnet-9-zzz --output-format json "x"  # negative control: must fail

modelUsage must name the exact id. Use the FULL id in agent_model_name, never sonnet/opus/fable: those aliases re-point at the newest model (2026-09-29 the harness's CLI ran sonnet→claude-sonnet-5, a newer CLI →claude-sonnet-5-5), so an alias row silently benchmarks a different model under its label. The agent harness runs the CLI found by executor.FindNativeBinary — ~/.local/bin/claude (self-updating) before any VSCode-bundled copy, which lags and once rejected a new model as unrecognized_model.

For Gemini models:

• Uses Vertex AI (not public API)
• Requires gcloud auth application-default login
• Check availability with verify_vertex_model.sh

2. Find Model Information

Search for official documentation:

# Find API model name and pricing
scripts/find_model_info.sh "GPT-5.1 API documentation pricing"

What to gather:

• Exact API model name (e.g., gpt-5.1 not GPT-5.1)
• Provider (openai, anthropic, google)
• Input price per 1K tokens
• Output price per 1K tokens
• Context limits (if relevant)
• Special features (adaptive reasoning, caching, etc.)

Reference: See resources/provider_endpoints.md

2a. Reasoning-Model Check (REQUIRED before gating any verdict)

The GLM-5.2 lesson (v0.30.0, 2026-07-19): GLM-5.2 was rejected in June as "worse than 5.1" — but the regression was OUR truncation: its always-on thinking phase (28–32K tokens) shared a 32,768 max_output_tokens budget with content, and the harness recorded neither reason_tokens nor finish_reason, so the guillotine was invisible. With 64K headroom, 5.2 beats 5.1 on every axis. Kimi K3 nearly repeated this (thinks by default, same 32K cap, unmeasured).

Before writing any gate verdict for a new model:

1. Probe default thinking — one small OpenRouter/API call, no reasoning params:

   # reasoning_tokens > 0 → the model thinks by default
   curl -s https://openrouter.ai/api/v1/chat/completions -H "Authorization: Bearer $OPENROUTER_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model":"<api_name>","messages":[{"role":"user","content":"Prove there are infinitely many primes, then state the 10th prime."}],"max_tokens":3000}' \
     | jq '{reasoning: .usage.completion_tokens_details.reasoning_tokens, provider}'
   

2. If it thinks: ground max_output_tokens in the DECLARED provider ceilings, not a guess. Context window ≠ output cap ≠ thinking budget — a "1M context" model may declare no completion cap at all (Kimi K3 via Moonshot: undeclared → generation is bounded only by request max_tokens + remaining window).

   # Per-provider documented ceilings for this model (machine-readable):
   curl -s "https://openrouter.ai/api/v1/models/<api_name>/endpoints" \
     | jq -r '.data.endpoints[] | "\(.provider_name): ctx=\(.context_length) max_completion=\(.max_completion_tokens // "undeclared")"'
   

   Providers DIVERGE (glm-5.2: 32768 on DeepInfra vs 131072 on Z.AI first-party vs 1M on others) — routing roulette means the effective cap varies per request unless you pin provider order. Set max_output_tokens from the first-party / dominant provider's declared ceiling, sized for thinking + content (never the fleet-default 32768), and record the provenance in the models.yml comment. Also check the vendor's own docs for the intended control surface: some models bound thinking by an effort dial (K3: Low/Standard/High/Max via reasoning_effort in models.yml), Anthropic by budget_tokens, not by output headroom. Reasoning bills as OUTPUT tokens — at K3's $15/M a chatty trace can cost more than the visible answer, so an eval that doesn't pin effort isn't reproducible; leave vendor-default unless deviating, and if you deviate, record it in models.yml (reasoning_effort).
3. Leave default thinking ON. A thinking-tuned model with thinking suppressed is not the model you're gating. The knob that is actually enforced is max_tokens headroom — OpenRouter third-party upstreams (Baidu, StreamLake) ignore reasoning: {max_tokens: N} (probed 2026-07-19); reasoning_max_tokens in models.yml is best-effort only.
4. Read finish_reason + reason_tokens in every failure before concluding capability (recorded on standard results since v0.30.0). A finish=length failure with huge reason_tokens is truncation, not weakness. Results banked before v0.30.0 cannot show this — never re-derive a verdict from them alone.
5. Cost note: thinking bills as output tokens. Budget/cost projections for a reasoning model must use output + reasoning, and expect ~2-10x the completion tokens of a non-thinking peer.

2b. Ollama Cloud models (flat-rate route)

Open-weight models at a flat subscription rate, as motoko-cloud-* rows. The row conventions, quota semantics and credit-rate measurement are in resources/ollama_cloud.md — read it before adding one.

3. Update models.yml

⛔ NEVER add an OpenRouter row for an Anthropic, OpenAI or Google (Gemini) model (Mark, 2026-09-29). Those vendors have direct lanes here (claude CLI subscription, codex subscription, Vertex). If the direct lane fails, fix or report the lane — do not add an or- twin to get a number. Open-weights releases (gemma, gpt-oss) are fine. Enforced by TestModels_FirstPartyVendorsNeverViaOpenRouter. Claude CLI rows pin the FULL id in agent_model_name, never sonnet/opus/fable (TestModels_ClaudeCLIRowsNeverUseAnAlias).

Add the model configuration:

# Add to models.yml
scripts/update_models_yml.sh \
  <friendly-name> \
  <api-name> \
  <provider> \
  <input-per-1k> \
  <output-per-1k>

Naming conventions:

• Friendly name: gpt5-1, claude-sonnet-4-5, gemini-3-pro
• API name: Exact string for API calls
• Use hyphens, lowercase

Also update:

• Model suites (benchmark_suite, extended_suite, dev_models)
• Add notes about special features
• Document agent CLI support (if available)

4. Run Test Benchmark

Verify end-to-end:

# Test with a simple benchmark
scripts/run_test_benchmark.sh <model-name>

What to verify:

• Benchmark completes successfully
• Results are reasonable (not garbage output)
• Token usage matches expectations
• Cost calculation works
• No errors in logs

5. Apply the Smoke-Test Gate (HARD RULE)

Rule of thumb (project-wide):

Rule out adding a model to our eval suite if it can't pass ALL the smoke tests.

The smoke test is the canonical smoke tier — benchmarks tagged tier: smoke in their YAML spec, selected with --tier smoke (NOT a hand-typed --benchmarks list). These are the fundamental "can it speak AILANG at all" tests that the established frontier tier passes cleanly. If a candidate fails them, the failure is on the model, not the harness or the benchmark.

The smoke tier is the source of truth — do NOT hardcode a benchmark list. Run ailang eval-suite --tier smoke --dry-run to see the current set (23 as of 2026-06-16, up from 17 — it grows, so always derive it, never trust this number). It includes fizzbuzz, adt_option, gcd_lcm, nested_records, record_update, recursion_fibonacci, type_safe_record_access, balanced_parens, etc. — all fundamental.

⚠️ csv_to_json_converter is tier: core, NOT tier: smoke. An earlier version of this skill hardcoded fizzbuzz,adt_option,csv_to_json_converter as "the smoke set." That was wrong: csv_to_json is a core-tier discriminator that the majority of frontier models fail in standard mode (gpt5 base, gemini-3-pro, gemini-3-flash, sonnet-4-5, gpt5-mini all FAIL it; only the top tier — opus-4-6/4-7, sonnet-4-6, gemini-3-1-pro, gpt5-2-codex/gpt5-4 — pass). Gating OS models on csv_to_json means "be top-3-tier or be cut," which unfairly excludes viable models. Keep csv_to_json in --tier core runs for ranking, never as an include/exclude gate. (Empirically verified against eval baselines 2026-06-02.)

Run smoke against a candidate (canonical tier):

Standard mode accepts --tier smoke directly. Agent mode requires an explicit --benchmarks list (deliberate cost guardrail), so derive it from the tier: tags — never hardcode (the list drifts):

# Derive the smoke set from the tier tags (works for both modes, stays in sync)
SMOKE=$(grep -l 'tier: smoke' benchmarks/*.yml | xargs -n1 basename | sed 's/\.yml$//' | paste -sd, -)

# Standard mode:
ailang eval-suite --models <candidate>,claude-sonnet-4-6 --tier smoke \
  --langs ailang --output /tmp/smoke_<candidate> --parallel 2

# Agent mode (must pass the derived list explicitly):
ailang eval-suite --agent --models <candidate>,claude-sonnet-4-6 \
  --benchmarks "$SMOKE" --langs ailang --output /tmp/smoke_<candidate> --parallel 2

# Tabulate pass/fail (agent mode → results land under /agent, standard → /standard)
for f in /tmp/smoke_<candidate>/*/*.json; do
  name=$(basename "$f" .json | sed 's/_[0-9]*$//')
  jq -r --arg name "$name" '"\($name)\t\(if .compile_ok and .runtime_ok and .stdout_ok then "PASS" else "FAIL" end)\t\(.err_code // .error_category // "—")"' "$f"
done | column -ts $'\t'

Decision tree (N = number of benchmarks in the smoke tier, 23 as of 2026-06-16 — derive it, don't assume):

1. claude-sonnet-4-6 fails any smoke-tier benchmark — smoke tier is broken; fix the benchmark (or its tier: tag) before evaluating candidates.
2. Candidate fails most of the tier — CUT. Do not add to models.yml. Note the failure types in the cut commit message (WRONG_LANG, syntax, runtime, wrong-output) for future reference.
3. Candidate fails 1–2 (near-clean) — NEAR-MISS. Optionally keep with a "near-miss" comment block in models.yml (precedent: motoko-or-gemma-4-26b, motoko-or-qwen3-5-35b-a3b). Re-run periodically; if it starts passing the tier clean, that's a signal stdlib/prompt has improved. Note: agent-mode failures with error_category: api_error + "step budget exhausted" are a harness step-budget cap, not a model gap — don't count them as capability failures (bump the motoko v2 step budget instead).
4. Candidate passes the tier clean — it has cleared the FLOOR, nothing more. Smoke qualifies a model; it does NOT rank it (see step 5.5). Add the opt-in models.yml entry now (gate PASSED, promotion PENDING), then go to step 5.5 to decide whether it actually earns a suite slot / replaces an incumbent.

Failure-mode taxonomy (worth capturing in the cut commit message):

Failure	Meaning	Likely cause
WRONG_LANG	Model produced Python/JS instead of AILANG	Prompt-following gap; small/MoE models lose plot at 23k-token system prompt
syntax-error (no WRONG_LANG)	Invented AILANG syntax (e.g. let rec, \n. lambda)	Model hasn't seen enough AILANG in training
wrong-output	Compiled and ran, wrong stdout	Spec-following gap, not language gap
runtime-error	Compiled, crashed at runtime	Logic bug

Reading smoke failures (full precedents, incl. the 2026-05-04 OS-model sweeps, in resources/smoke_precedents.md):

• api_error is infrastructure (rate limit, timeout, harness) — re-run before counting it; "exceeded hard timeout" on an agent run is usually the benchmark's timeout:, not the model.
• compile_error = invented syntax; WRONG_LANG = produced Python/JS — genuine model gaps.
• Agent mode is a separate gate: a standard-mode failure may pass agent smoke, and vice versa.
• New OpenRouter vendor prefix → add it to internal/ai/config.go::openrouterVendorPrefixes.

5.5 Smoke is a FLOOR, not a RANKING — place the candidate on the ANCHORED ELO series (HARD RULE)

Passing smoke is necessary but NOT sufficient. Smoke says "this model can speak AILANG at all"; it does NOT say "this model is good enough to add" or "this model beats the incumbent." Those are RANKING questions, and smoke is saturated — every frontier-class model scores ~the same on it. Never make an add/keep/replace decision on smoke numbers.

Why: the smoke tier is deliberately fundamental ("can it speak AILANG"), so any viable model passes all of it. A smoke tie is the expected outcome, not a signal — it carries zero ranking information. Place on the HARD set: --tier long-frontier,stretch,frontier with --trials 3 (31 standard-mode benchmarks).

Measured 2026-10-19 (DeepSeek V4.1 Flash 1017 vs 0731): --tier smoke,frontier tied 25/26 vs 25/26 (ELO 2225 vs 2228) — frontier alone holds only 3 benchmarks that run in standard mode (docx_/markdown_reimplement are agent-only), and core is saturated for frontier-class models too. The hardest anchored benchmarks (gauntlet_10 2241, commonmark_emphasis, quine, legal_obligation_engine) were moved to long-frontier on 2026-10-06 for LOCAL AGENT timeouts — a reason that does not apply to a standard-mode API call. The hard set separated the pair: 80/93 vs 84/93, ELO 2042 vs 2128, with per-benchmark discordance to read.

Cost: compare BILLED, not banked. Banked cost = registry list price × tokens, but sort=throughput routes to whichever host is fastest, not the list-price host. That run banked $0.87 vs $0.55 (1.6x) while OpenRouter billed $0.0077 vs $0.0082 per call — a tie. Read gen_ai.usage.total_cost per model from the Broadcast spans (CLAUDE.md §1) before calling one model dearer.

⛔ Run the CANDIDATE ALONE. Do not re-run incumbents or anchors.

M-EVAL-ROLLING-ELO (landed 2026-08-27, PRs #939/#942) changed this. ELO ratings used to be incomparable across fits — FitFromTrials seeded every model and benchmark at 1500 with no scale anchor, so the same rows produced different absolute numbers in different pools (measured: or-glm-5-3-flash rated 2763 in one pool vs 1995 in another over comparable rows). That is why the old protocol re-ran candidate + incumbent + anchor together — a shared pool was the only way to make numbers mean anything.

That is no longer true, and doing it now is pure waste:

• internal/eval_harness/anchor_v1.json freezes the fitted difficulties of the discriminating standard benchmarks. Standard-mode fits hold that panel fixed and let model ratings move (`cmd/ailang/eval_elo.go:172`).
• So a candidate run alone is placed onto the same scale as every model ever measured. Anchored drift is 31.2 ELO vs 311.7 unanchored.
• D3 retired full baselines as the default release measurement. The full run is demoted to quarterly re-anchoring (make eval-baseline FULL=true).

# CORRECT — candidate only; the anchored fit places it against banked history.
ailang eval-suite --models <candidate> --tier long-frontier,stretch,frontier \
  --trials 3 --langs ailang --output /tmp/cf_<candidate> --parallel 6

ailang eval-elo /tmp/cf_<candidate> --json     # read-only ANCHORED placement fit
go run ./tools/eval-elo /tmp/cf_<candidate> --mode standard \
  --persist ~/.ailang/state/observatory.db              # bank it into the series

ailang eval-elo does NOT persist — it refuses --persist and tells you to use tools/eval-elo. Persisting through the cmd path silently no-ops (it swallowed --persist for weeks; see project_agent_ratings_seeding_evalelo).

Compare the resulting rating to the banked ratings already in observatory.db (LoadModelRatings) — that is what the series is for.

⚠️ CHECK THE BANKED SERIES IS ACTUALLY ANCHORED BEFORE COMPARING. An anchored placement and a pre-anchor banked rating are on DIFFERENT SCALES, and nothing in the output warns you. Measured 2026-09-01 on the dev box: every model_ratings row for mode='standard' was stamped 2026-08-03 — before the anchor landed (2026-08-28) — so those 19 values are unanchored, and trial_history held agent-mode rows only (739 rows, 3 models), meaning there were no banked standard trials to re-level them from. Reading Hy4's anchored 1915.5 against that table would have been exactly the 2763-vs-1995 error the anchor exists to prevent.

sqlite3 ~/.ailang/state/observatory.db \
  "SELECT model_id, ROUND(rating,1), n_trials, substr(last_updated,1,10)
     FROM model_ratings WHERE mode='standard' ORDER BY rating DESC;"
sqlite3 ~/.ailang/state/observatory.db \
  "SELECT mode, COUNT(*), COUNT(DISTINCT model_id) FROM trial_history GROUP BY mode;"

Update 2026-09-29: anchored standard rows now exist — claude-opus-5-5 2871.9 (floor, 29/29), gpt6-sol 2288.7, gpt6-luna 2008.2 (all 2026-09-22, 29-benchmark core+frontier panel) and claude-sonnet-5-5 ≈2296 (28/29). The check below is still how you tell which rows qualify.

If last_updated predates the anchor, or trial_history has no rows for the mode you are placing in, you have a placement but no valid comparison set. Say so plainly rather than ranking against stale numbers. Report the candidate's pass profile against the anchored benchmark difficulties (which the fit does give you) and treat the leaderboard position as unavailable until the series is re-fit. Do NOT "fix" this by re-running comparators — that is the anti-pattern above; the fix is banking standard-mode trials so the series can accumulate.

When you DO still co-run models

Three cases, and only these:

1. Agent mode. There is no agent-mode anchor yet — eval_elo.go:171-172 anchors standard mode only, agent fits are unanchored. For an agent-mode ranking question the old same-pool rule still holds.
2. The incumbent's banked rating predates a baseline-moving language change. The anchor pins benchmark difficulty, not the harness or stdlib. A change that moves what models can do (e.g. the 2026-07-29 extension fix) invalidates older per-benchmark rates. Check the banked rating's version/date provenance first; if it is stale, re-run just the incumbent — never the whole panel.
3. A paired/discordant analysis, where ailang eval-paired <on> <off> needs both arms from the same run by construction.

When you genuinely do run several models, they must still go in ONE eval-suite command — it overwrites its output directory (.claude/rules/eval.md). That rule is about not clobbering results; it is not a reason to add models.

Deciding promote / replace

• Match-or-beat the incumbent to promote. A tie at higher cost keeps the incumbent; a tie at equal-or-lower cost favours the newer generation.
• Close call on N=1 — core is ~26 single-shot runs and OS-model variance is real. Within 1–2 benchmarks (or overlapping ELO bands), escalate to N≥3 before deciding. Never flip an incumbent on a 1-benchmark N=1 delta.
• Read finish_reason + reason_tokens on every failure before calling it capability (§2a). A finish=length with large reasoning is truncation.

⚠️ Anti-pattern (2026-06-16, GLM-5.2 vs GLM-5.1): GLM-5.2 cleared standard smoke at 22/23 — an exact tie with GLM-5.1 (both failed only dense_operator_program, which the anchor ALSO failed → a benchmark/harness issue, not a model gap). The first-pass conclusion was "tie at +43% cost → keep GLM-5.1." That was WRONG. A smoke tie is meaningless because smoke is saturated — it proves only that the candidate cleared the floor and QUALIFIES. When a candidate ties the incumbent on smoke, that is your cue to run core, NOT your answer.

⚠️ Anti-pattern (2026-09-01, Hy4 preview): after Hy4 passed smoke 23/23, the placement run was launched as six models × 31 benchmarks = 186 runs, ~$3.75 — candidate plus incumbent plus four comparators, on the pre-rolling-ELO reflex that a shared pool was needed. It was not: five of those six models already had banked anchored ratings, and the candidate alone costs $0.23. Killed at $0.24. A comparator you re-run is a comparator you pay for twice.

6. Document the Model

Update relevant documentation:

• Add model to this skill's resource guide
• Note any special parameters (e.g., max_completion_tokens for GPT-5.1)
• Document authentication requirements
• Add to teaching prompts if needed

7. Bank the placement — do NOT run a full baseline

Once the candidate has its core/frontier placement, persist it into the anchored series so the next question can be answered from banked data instead of a re-run:

go run ./tools/eval-elo /tmp/cf_<candidate> --mode standard --persist ~/.ailang/state/observatory.db

--persist takes the DB path, not the results dir. database is locked means another ailang process (coordinator, a running eval) holds observatory.db — retry once it is idle.

make eval-baseline FULL=true is NOT part of adding a model. D3 of M-EVAL-ROLLING-ELO demoted the full baseline to a quarterly re-anchoring event (and longitudinal spot-checks). Running one to place a new model costs $5-25 + hours of wall clock to produce a number the anchored fit already gives you for well under a dollar. If you think you need a full baseline, you almost certainly need a linking run instead — re-read §5.5.

Resources

Provider Endpoints

See resources/provider_endpoints.md for:

• API endpoint URLs for each provider
• Authentication methods
• How to test access manually
• Common errors and fixes

Pricing Guide

See resources/pricing_guide.md for:

• How to find official pricing
• Price conversion (per 1M → per 1K)
• Cost calculation verification
• Caching and discounts

Progressive Disclosure

This skill loads information progressively:

1. Always loaded: This SKILL.md file (workflow and script descriptions)
2. Execute as needed: Scripts in scripts/ (testing, updating, verification)
3. Load on demand: Resources (detailed endpoint docs, pricing references)

Notes

Important:

• Always test API access BEFORE updating models.yml
• Vertex AI (Gemini) requires gcloud auth, not API key
• GPT-5.1+ uses max_completion_tokens instead of max_tokens
• New models may not be available in all regions immediately
• Check for preview/beta status before adding to production suites

Prerequisites:

• API keys: . ~/.config/ailang/secrets.env (see §1 — shells do not source it)
• For Gemini: gcloud CLI installed and authenticated
• For Ollama Cloud: local ollama daemon running + ollama signin done (inference); OLLAMA_API_KEY set only for the /api/usage quota gauge
• For Gemini: GCP project set (gcloud config set project PROJECT_ID)
• curl, python3, and jq available in PATH

Files modified by this skill:

• internal/modelreg/models.yml - Model configurations (the embedded registry)
• Anthropic rows also need, per model: internal/ai/reasoning_anthropic.go (thinking style — CanDisable:false if {type:"disabled"} 400s), internal/ai/anthropic/cache.go (min cacheable prefix), ceilingLimited in internal/eval_harness/models_headroom_test.go, and anthropicCacheReadMultiplier in internal/modelreg/resolve_test.go if cache reads ≠ 0.1x
• resources/provider_endpoints.md - When adding a new provider (e.g. Ollama Cloud)
• (Optional) prompts/vX.Y.Z.md - Teaching prompts
• (Optional) .claude/skills/model-manager/resources/ - Local model database