CLI E2E Testing

Tests use BATS (Bash Automated Testing System) located in e2e/.

Quick Start

# Run all tests
./e2e/run.sh

# Run specific test file
./e2e/test/libs/bats/bin/bats e2e/tests/02-commands/t03-volumes.bats

Test File Template

#!/usr/bin/env bats

load '../../helpers/setup'

setup() {
    # Use unique names with timestamp to avoid conflicts
    export TEST_DIR="$(mktemp -d)"
    export RESOURCE_NAME="e2e-test-$(date +%s)"
}

teardown() {
    # Always clean up temp directories
    [ -n "$TEST_DIR" ] && [ -d "$TEST_DIR" ] && rm -rf "$TEST_DIR"
}

@test "descriptive test name" {
    run $CLI_COMMAND subcommand args
    assert_success
    assert_output --partial "expected text"
}

Assertions

# Exit status
assert_success                    # exit code 0
assert_failure                    # exit code != 0

# Output matching
assert_output --partial "text"    # output contains text
refute_output --partial "text"    # output does NOT contain text
assert_output --regexp "pattern"  # output matches regex

# Line matching
assert_line --index 0 "first line"

Key Patterns

1. Unique Resource Names

# Always use timestamp to prevent test conflicts
export VOLUME_NAME="e2e-volume-$(date +%s)"
export ARTIFACT_NAME="e2e-artifact-$(date +%s)"

2. Extended Timeouts for CI

# CI environments are slower - use 120s timeout
run $CLI_COMMAND run agent-name \
    --artifact-name "$ARTIFACT_NAME" \
    --timeout 120 \
    "echo hello"

3. Debug Output with Echo Comments

@test "multi-step test" {
    echo "# Step 1: Setup..."
    # ... setup code ...

    echo "# Step 2: Execute..."
    run $CLI_COMMAND ...

    echo "# Step 3: Verify..."
    assert_success
}

4. Extract IDs from Output

# Extract UUID patterns
CHECKPOINT_ID=$(echo "$output" | grep -oP 'Checkpoint:\s*\K[a-f0-9-]{36}' | head -1)
SESSION_ID=$(echo "$output" | grep -oP 'Session:\s*\K[a-f0-9-]{36}' | head -1)

# Verify extraction succeeded
[ -n "$CHECKPOINT_ID" ] || {
    echo "# Failed to extract checkpoint ID"
    echo "$output"
    return 1
}

5. Test Both Success and Failure

@test "valid input succeeds" {
    run $CLI_COMMAND volume init
    assert_success
}

@test "invalid input fails with error" {
    run $CLI_COMMAND volume pull "nonexistent"
    assert_failure
    assert_output --partial "not found"
}

6. Suppress Output for Setup Commands

# Use >/dev/null for setup commands that must succeed
$CLI_COMMAND artifact init >/dev/null
$CLI_COMMAND artifact push >/dev/null

# Only use `run` when you need to check output/status
run $CLI_COMMAND artifact push
assert_success

File Organization

e2e/
├── tests/
│   ├── 01-smoke/          # Basic CLI availability tests
│   │   └── t01-smoke.bats
│   └── 02-commands/       # Feature-specific tests
│       ├── t01-validation.bats
│       ├── t03-volumes.bats
│       └── t04-vm0-artifact-checkpoint.bats
├── fixtures/
│   └── configs/           # Test agent configurations
└── helpers/
    └── setup.bash         # Shared setup (loads bats-assert)

Naming Convention

• Test files: tXX-feature-name.bats
• Test resources: e2e-{type}-$(date +%s)

CI Integration

Tests run with parallel file execution:

bats -j 4 --no-parallelize-within-files tests/**/*.bats

• -j 4: Run up to 4 test files in parallel
• --no-parallelize-within-files: Tests within a file run sequentially

Checklist

Before submitting:

• Uses unique resource names with timestamp
• Has setup() and teardown() for cleanup
• Uses --timeout 120 for vm0 run commands
• Tests both success and failure cases
• Includes debug echo comments for multi-step tests
• Runs successfully: ./e2e/run.sh tests/02-commands/your-test.bats