Claude Code Plugins

Community-maintained marketplace

Feedback

schema-e2e-validation

@terrylica/cc-skills
1
0

>

Install Skill

1Download skill
2Enable skills in Claude

Open claude.ai/settings/capabilities and find the "Skills" section

3Upload to Claude

Click "Upload skill" and select the downloaded ZIP file

Note: Please verify skill by going through its instructions before using it.

SKILL.md

name schema-e2e-validation
description Run Earthly E2E validation for YAML schema contracts. Use when validating YAML schema changes, testing schema contracts against live ClickHouse, or regenerating Python types, DDL, and docs from YAML. For SQL schema design and optimization, use clickhouse-architect skill instead.
allowed-tools Read, Bash, Grep

Schema E2E Validation

When to Use

  • Validating schema changes before commit
  • Verifying YAML schema matches live ClickHouse Cloud
  • Regenerating Python types, DDL, or docs
  • Running full schema workflow validation

Prerequisites

Docker Runtime (Required)

Earthly requires Docker. Start Colima before running:

colima start

Check if running:

docker ps  # Should not error

Doppler Access (For validation targets)

Required for +test-schema-validate and +test-schema-e2e:

doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd

Earthly Installation

brew install earthly

Quick Commands

Generation only (no secrets)

cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
earthly +test-schema-generate

Full E2E with validation (requires Doppler)

cd /Users/terryli/eon/gapless-network-data
colima start  # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e

All non-secret targets

cd /Users/terryli/eon/gapless-network-data
earthly +all

Artifacts

After running +test-schema-generate or +test-schema-e2e, check ./earthly-artifacts/:

Path Contents
types/blocks.py Pydantic + TypedDict models
types/__init__.py Package init
ddl/ethereum_mainnet.sql ClickHouse DDL
docs/ethereum_mainnet.md Markdown documentation

For E2E, artifacts are under e2e/types/, e2e/ddl/, e2e/docs/.

Earthfile Targets Reference

Target Secrets Purpose
+deps No Install uv + dependencies
+build No Copy source files
+test-unit No Run pytest
+test-schema-generate No Generate types/DDL/docs
+test-schema-validate Yes Validate vs ClickHouse
+test-schema-e2e Yes Full workflow + artifacts
+all No Run all non-secret targets

Troubleshooting

"could not determine buildkit address - is Docker or Podman running?"

Cause: Docker/Colima not running

Fix:

colima start
# Wait for "done" message, then retry
earthly +test-schema-generate

"unable to parse --secret-file argument"

Cause: Wrong flag name or malformed secrets file

Fix: The correct flag is --secret-file-path (NOT --secret-file). The wrapper script handles this, but if running manually:

# WRONG
earthly --secret-file=/path/to/secrets +target

# CORRECT
earthly --secret-file-path=/path/to/secrets +target

Also ensure secrets file has no quotes around values:

# WRONG format
CLICKHOUSE_HOST="host.cloud"

# CORRECT format
CLICKHOUSE_HOST=host.cloud

"OSError: Readme file does not exist: README.md"

Cause: hatchling build backend requires README.md in container

Fix: Ensure Earthfile copies README.md in deps target:

deps:
    COPY pyproject.toml uv.lock README.md ./  # README.md required!

"missing secret" during validation

Cause: Doppler not configured or secrets not passed

Fix:

# Verify Doppler has the secrets
doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE

# Use the wrapper script (handles secret injection)
./scripts/earthly-with-doppler.sh +test-schema-validate

Cache Issues

Force rebuild without cache:

earthly --no-cache +test-schema-e2e

Implementation Details

Doppler Secret Injection

The wrapper script scripts/earthly-with-doppler.sh:

  1. Downloads secrets from Doppler
  2. Filters for CLICKHOUSE_* variables
  3. Strips quotes (Doppler outputs KEY="value", Earthly needs KEY=value)
  4. Passes via --secret-file-path flag
  5. Cleans up temp file on exit

Secrets Required

Secret Purpose
CLICKHOUSE_HOST_READONLY ClickHouse Cloud host
CLICKHOUSE_USER_READONLY Read-only user
CLICKHOUSE_PASSWORD_READONLY Read-only password

Related Files

File Purpose
/Users/terryli/eon/gapless-network-data/Earthfile Main build file
/Users/terryli/eon/gapless-network-data/scripts/earthly-with-doppler.sh Secret injection wrapper
/Users/terryli/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml SSoT schema
/Users/terryli/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md ADR

Validation History

  • 2025-12-03: Created and validated with full E2E run against ClickHouse Cloud
  • Lessons Learned:
    • --secret-file-path not --secret-file (Earthly v0.8.16)
    • Doppler --format env outputs quotes, must strip with sed 's/"//g'
    • README.md must be copied for hatchling build backend
    • Colima must be started before Earthly runs

Design Authority

This skill validates schemas but does not design them. For schema design guidance (ORDER BY, compression, partitioning), invoke quality-tools:clickhouse-architect first.

Related Skills

Skill Purpose
quality-tools:clickhouse-architect Schema design before validation
devops-tools:clickhouse-cloud-management Cloud credentials for E2E tests
devops-tools:clickhouse-pydantic-config Client configuration