Test Runner for Worktrees

Guide for running integration tests properly in both git worktrees and the main repository, with emphasis on environment variable sourcing and working directory context.

Overview

During development in git worktrees, running tests requires careful attention to:

1. Environment variables - Credentials may already be loaded from devcontainer
2. Working directory - Tests may behave differently in main vs worktree due to SDK version differences
3. Context awareness - Understanding whether you're in main workspace or a feature branch worktree

This skill codifies the lessons learned from issue #186 to prevent future confusion and wasted time.

Key Principles

1. Always Check Environment Variables First

Problem: Repeatedly asking users for credentials when they're already available in the environment.

Solution: Check if environment variables are set BEFORE asking the user.

# ✅ CORRECT: Check without exposing values
[ -n "$LANGSMITH_API_KEY" ] && echo "LANGSMITH_API_KEY is set" || echo "LANGSMITH_API_KEY not set"
[ -n "$ANTHROPIC_API_KEY" ] && echo "ANTHROPIC_API_KEY is set" || echo "ANTHROPIC_API_KEY not set"

If not set, source from devcontainer:

# Source devcontainer environment file
source /workspace/.devcontainer/.env

# Verify variables are now set
[ -n "$LANGSMITH_API_KEY" ] && echo "LANGSMITH_API_KEY is set" || echo "LANGSMITH_API_KEY not set"

Never expose actual values:

# ❌ WRONG: Exposes sensitive data
echo "LANGSMITH_API_KEY=$LANGSMITH_API_KEY"

# ✅ CORRECT: Checks without exposing
[ -n "$LANGSMITH_API_KEY" ] && echo "Set" || echo "Not set"

2. Always Verify Current Working Directory

Problem: Running tests from the wrong directory leads to unexpected failures and confusion.

Key Difference: Tests run from worktrees use the feature branch SDK, while tests from main workspace use the main branch SDK.

Example from #186:

• Background tests ran from /workspace (main branch) → FAILED (old SDK without Queued enum)
• Worktree tests ran from /workspace/wip/codekiln-186-test-helpers → PASSED (new SDK with Queued from PR #185)

Solution: Always verify working directory before running tests.

# ✅ CORRECT: Verify and navigate to worktree
pwd
cd /workspace/wip/<worktree-name>

# Run tests from worktree
cargo test --test integration_test -- --ignored --nocapture

3. Understand Test Types and Their Requirements

Different tests have different requirements and durations:

Unit Tests (No API Required):

# Fast (<1 second), no credentials needed
cargo test --test integration_deployment_workflow test_deployment_url_extraction -- --nocapture

Helper Tests (API Required, 1-5 seconds):

# Require credentials, fast feedback
cargo test --test integration_deployment_workflow test_list_github_integrations -- --ignored --nocapture
cargo test --test integration_deployment_workflow test_list_github_repositories -- --ignored --nocapture
cargo test --test integration_deployment_workflow test_find_integration_for_repo -- --ignored --nocapture

Full Workflow Tests (5-30 minutes):

# Long-running, creates real resources
cargo test --test integration_deployment_workflow test_deployment_workflow -- --ignored --nocapture

Workflows

Workflow 1: Run Tests in Worktree (Recommended)

When working on a feature branch in a worktree, always run tests from the worktree directory.

Steps:

# Step 1: Navigate to worktree
cd /workspace/wip/<worktree-name>

# Step 2: Verify you're in the correct location
pwd
# Should output: /workspace/wip/<worktree-name>

# Step 3: Check environment variables (without exposing them)
[ -n "$LANGSMITH_API_KEY" ] && echo "LANGSMITH_API_KEY is set" || echo "LANGSMITH_API_KEY not set"

# Step 4: Source devcontainer .env if needed
if [ -z "$LANGSMITH_API_KEY" ]; then
  source /workspace/.devcontainer/.env
  echo "Sourced environment from /workspace/.devcontainer/.env"
fi

# Step 5: Run tests
# For unit tests (no API)
cargo test --test integration_deployment_workflow test_deployment_url_extraction -- --nocapture

# For helper tests (fast, require API)
cargo test --test integration_deployment_workflow test_list_github_integrations -- --ignored --nocapture

# For full workflow tests (slow, require API)
cargo test --test integration_deployment_workflow test_deployment_workflow -- --ignored --nocapture

Why this matters:

• Worktree has the feature branch code
• Feature branch may have SDK changes not in main
• Tests will use the correct SDK version

Workflow 2: Run Tests in Main Workspace

When testing against the main/release branch SDK, run from main workspace.

Steps:

# Step 1: Navigate to main workspace
cd /workspace

# Step 2: Verify branch
git branch --show-current
# Should output: main (or release/vX.Y.Z)

# Step 3: Check environment variables
[ -n "$LANGSMITH_API_KEY" ] && echo "LANGSMITH_API_KEY is set" || echo "LANGSMITH_API_KEY not set"

# Step 4: Run tests
cargo test --workspace --all-features

Workflow 3: Pre-Commit Testing

Before committing, run all checks from the worktree.

Steps:

# From worktree directory
cd /workspace/wip/<worktree-name>

# Run pre-commit checklist
cargo fmt
cargo check --workspace --all-features
cargo clippy --workspace --all-features -- -D warnings
cargo test --workspace --all-features
cargo fmt --check

Why from worktree:

• Tests your feature branch changes
• Catches issues before CI
• Prevents wasted CI cycles

Common Mistakes to Avoid

❌ Mistake 1: Asking User for Credentials When Already Set

Wrong approach:

# Don't ask user to provide credentials without checking first
echo "Please provide your LANGSMITH_API_KEY"

Correct approach:

# Check first
if [ -z "$LANGSMITH_API_KEY" ]; then
  # Try sourcing from devcontainer
  source /workspace/.devcontainer/.env

  # If still not set, then ask
  if [ -z "$LANGSMITH_API_KEY" ]; then
    echo "LANGSMITH_API_KEY not found. Please set it in /workspace/.devcontainer/.env"
  fi
fi

❌ Mistake 2: Running Tests from Wrong Directory

Wrong approach:

# Running from main workspace when working on feature branch
cd /workspace
cargo test --test integration_deployment_workflow -- --ignored
# May fail due to old SDK version in main branch

Correct approach:

# Always verify pwd first
pwd
cd /workspace/wip/codekiln-186-test-helpers
cargo test --test integration_deployment_workflow -- --ignored

❌ Mistake 3: Exposing Sensitive Environment Variables

Wrong approach:

# Never expose actual values
echo "LANGSMITH_API_KEY=$LANGSMITH_API_KEY"
echo "Using key: ${LANGSMITH_API_KEY}"

Correct approach:

# Only check if set, never show value
[ -n "$LANGSMITH_API_KEY" ] && echo "LANGSMITH_API_KEY is set" || echo "LANGSMITH_API_KEY not set"

❌ Mistake 4: Not Understanding Test Context

Wrong approach:

# Running background tests from main workspace while developing in worktree
cd /workspace
cargo test --test integration_test -- --ignored &
# This will test old SDK from main, not your changes

Correct approach:

# Run from worktree to test your changes
cd /workspace/wip/<worktree-name>
cargo test --test integration_test -- --ignored

Environment Variable Reference

Required Variables for Integration Tests

LangSmith API:

• LANGSMITH_API_KEY - API key for LangSmith authentication
• LANGSMITH_ORGANIZATION_ID - Organization ID (optional, auto-detected)

Anthropic API:

• ANTHROPIC_API_KEY - API key for Claude models

Checking Variables (Template)

# Check all required variables at once
echo "Checking required environment variables..."
[ -n "$LANGSMITH_API_KEY" ] && echo "✓ LANGSMITH_API_KEY is set" || echo "✗ LANGSMITH_API_KEY not set"
[ -n "$ANTHROPIC_API_KEY" ] && echo "✓ ANTHROPIC_API_KEY is set" || echo "✗ ANTHROPIC_API_KEY not set"

# Source from devcontainer if any are missing
if [ -z "$LANGSMITH_API_KEY" ] || [ -z "$ANTHROPIC_API_KEY" ]; then
  echo "Sourcing from /workspace/.devcontainer/.env..."
  source /workspace/.devcontainer/.env

  # Check again
  [ -n "$LANGSMITH_API_KEY" ] && echo "✓ LANGSMITH_API_KEY is set" || echo "✗ LANGSMITH_API_KEY not set"
  [ -n "$ANTHROPIC_API_KEY" ] && echo "✓ ANTHROPIC_API_KEY is set" || echo "✗ ANTHROPIC_API_KEY not set"
fi

Test Execution Templates

Template 1: Run Single Test

# Navigate to worktree
cd /workspace/wip/<worktree-name>

# Verify location
pwd

# Check environment
[ -n "$LANGSMITH_API_KEY" ] && echo "LANGSMITH_API_KEY is set" || source /workspace/.devcontainer/.env

# Run specific test
cargo test --test <test_file> <test_name> -- --ignored --nocapture

Template 2: Run All Helper Tests

# Navigate to worktree
cd /workspace/wip/<worktree-name>

# Check environment
[ -n "$LANGSMITH_API_KEY" ] && echo "LANGSMITH_API_KEY is set" || source /workspace/.devcontainer/.env

# Run all integration tests (fast ones)
cargo test --test integration_deployment_workflow test_list_ -- --ignored --nocapture

Template 3: Run Long-Running Test in Background

# Navigate to worktree
cd /workspace/wip/<worktree-name>

# Check environment
[ -n "$LANGSMITH_API_KEY" ] && echo "LANGSMITH_API_KEY is set" || source /workspace/.devcontainer/.env

# Run test in background
cargo test --test integration_deployment_workflow test_deployment_workflow -- --ignored --nocapture > test_output.log 2>&1 &

# Save PID
echo $! > test_pid.txt

# Monitor progress
tail -f test_output.log

Troubleshooting

Tests Fail with "API key not found"

Symptom: Tests fail immediately with authentication errors.

Diagnosis:

# Check if env vars are set
[ -n "$LANGSMITH_API_KEY" ] && echo "Set" || echo "Not set"

Solution:

# Source from devcontainer
source /workspace/.devcontainer/.env

# Verify
[ -n "$LANGSMITH_API_KEY" ] && echo "Set" || echo "Not set"

Tests Fail with "type/enum not found"

Symptom: Tests fail with compilation errors about missing types or enum variants.

Diagnosis:

# Check current location
pwd
# If output is /workspace, you're in main workspace (may have old SDK)

Solution:

# Navigate to worktree with feature branch
cd /workspace/wip/<worktree-name>

# Run tests from worktree
cargo test --test integration_test -- --ignored --nocapture

Tests Pass in Worktree, Fail in CI

Symptom: Tests work locally in worktree but fail when pushed to CI.

Common Causes:

1. Feature branch depends on another PR not yet merged
2. SDK changes in worktree not compatible with main branch

Diagnosis:

# Test from main workspace to simulate CI environment
cd /workspace
git checkout main
git pull origin main
cargo test --workspace --all-features

Solution:

• Ensure prerequisite PRs are merged first
• Update feature branch to include necessary changes

Background Tests Show Different Results

Symptom: Running tests in background from main workspace shows failures, but running same test from worktree passes.

Cause: Background tests are using main branch SDK, not feature branch SDK.

Solution:

# Always run from worktree when testing feature branch
cd /workspace/wip/<worktree-name>
cargo test --test integration_test -- --ignored --nocapture

Security Best Practices

Never Expose Credentials

✅ CORRECT:

# Check without exposing
[ -n "$LANGSMITH_API_KEY" ] && echo "Set" || echo "Not set"

# Use in commands without echoing
cargo test --test integration_test -- --ignored --nocapture

❌ WRONG:

# Never do these
echo $LANGSMITH_API_KEY
echo "Key: $LANGSMITH_API_KEY"
env | grep LANGSMITH

Store Credentials Securely

Best practices:

• Store in /workspace/.devcontainer/.env (gitignored)
• Never commit .env files
• Use placeholders in documentation: <your-api-key>

Integration with Other Tools

With git-worktrees Skill

After creating a worktree, use this skill to run tests:

# Create worktree (from git-worktrees skill)
git worktree add -b codekiln/186-test-helpers wip/codekiln-186-test-helpers main

# Navigate and run tests (from this skill)
cd wip/codekiln-186-test-helpers
[ -n "$LANGSMITH_API_KEY" ] && echo "Set" || source /workspace/.devcontainer/.env
cargo test --test integration_test -- --ignored --nocapture

With Pre-Commit Checklist

Before committing, run the pre-commit checklist from worktree:

cd /workspace/wip/<worktree-name>
cargo fmt && \
cargo check --workspace --all-features && \
cargo clippy --workspace --all-features -- -D warnings && \
cargo test --workspace --all-features && \
cargo fmt --check

Quick Reference

Environment Check One-Liner

[ -n "$LANGSMITH_API_KEY" ] && echo "✓ LANGSMITH_API_KEY set" || (source /workspace/.devcontainer/.env && [ -n "$LANGSMITH_API_KEY" ] && echo "✓ LANGSMITH_API_KEY set after source" || echo "✗ LANGSMITH_API_KEY not found")

Worktree Test One-Liner

cd /workspace/wip/<worktree-name> && [ -n "$LANGSMITH_API_KEY" ] || source /workspace/.devcontainer/.env && cargo test --test integration_test -- --ignored --nocapture

Complete Test Flow

# 1. Verify location
pwd

# 2. Navigate to worktree if needed
cd /workspace/wip/<worktree-name>

# 3. Check and source environment
[ -n "$LANGSMITH_API_KEY" ] || source /workspace/.devcontainer/.env

# 4. Run tests
cargo test --test integration_test -- --ignored --nocapture

Related Documentation

• Issue #186 - Where these patterns were discovered
• git-worktrees skill - For creating and managing worktrees
• Pre-Commit Checklist - @docs/dev/README.md "Pre-Commit Checklist" section
• GitHub Workflow - @docs/dev/github-workflow.md

Key Takeaways

1. Always check environment variables before asking user
2. Always verify pwd before running tests
3. Run tests from worktree when working on feature branch
4. Never expose sensitive environment variable values
5. Understand test context (main workspace vs worktree)

These principles save time, prevent confusion, and improve security.