jq-transforms

Overview

jq-transforms is a schema-driven JSON transformation library built specifically for AI agents. It provides 60+ composable transformations for JSON manipulation, each with clear input/output contracts, comprehensive tests, and minimal implementations. The library emphasizes discoverability through schemas and composability through Unix pipes.

When to Use This Skill

Use this skill when:

• Transforming JSON data structures (CRUD operations, nested paths)
• Filtering or mapping arrays within JSON objects
• Extracting specific fields or patterns from JSON
• Validating JSON schemas or field types
• Converting between JSON types
• Composing multi-step JSON transformations
• Processing API responses or configuration files

Core Transformation Categories

CRUD Operations (crud-*)

Path-based object manipulation for getting, setting, deleting, checking existence, merging, and querying nested JSON structures.

Common examples:

• crud-get: Retrieve value at nested path with optional default
• crud-set: Set value at path, creating intermediate objects
• crud-delete: Remove field at path
• crud-has: Check if path exists
• crud-merge: Recursively merge two objects

Array Operations (array-*)

Array transformations including filtering, mapping, reducing, flattening, chunking, and deduplication.

Common examples:

• array-filter: Filter array by field value
• array-map: Extract field from each array element
• array-reduce: Aggregate array (sum, avg, min, max, count)
• array-unique: Remove duplicate values
• array-flatten: Flatten nested arrays to specified depth

String Operations (string-*)

String manipulation within JSON objects.

Common examples:

• string-split: Split string into array by separator
• string-join: Join array into string with separator
• string-replace: Replace pattern in string
• string-trim: Remove whitespace from string

Extraction (extract-*)

Pattern-based extraction from text fields.

Common examples:

• extract-urls: Extract URLs from text
• extract-code-blocks: Extract code blocks from markdown
• extract-mentions: Extract @mentions from text

Validation (validate-, is-, has-*)

Field validation and type checking.

Common examples:

• validate-required: Check required fields exist
• validate-types: Validate field types
• is-timestamp: Check if value is valid timestamp
• has-field: Check if nested path exists

Type Conversions (to-*)

Convert between JSON types safely.

Common examples:

• to-string, to-number, to-boolean, to-array, to-object

Usage Pattern

All transformations follow a consistent invocation pattern:

jq -f <transform-dir>/transform.jq [--arg key val] [--argjson key json] input.json

Example - Get nested value:

echo '{"user":{"name":"skogix"}}' | \
  jq -f crud-get/transform.jq --arg path "user.name"
# Output: "skogix"

Example - Filter array:

echo '{"items":[{"status":"active"},{"status":"inactive"}]}' | \
  jq -f array-filter/transform.jq \
    --arg array_field "items" \
    --arg field "status" \
    --arg value "active"
# Output: {"items":[{"status":"active"}]}

Example - Compose multiple transforms:

cat data.json | \
  jq -f array-filter/transform.jq --arg array_field "users" --arg field "active" --arg value "true" | \
  jq -f array-map/transform.jq --arg array_field "users" --arg field "email"

Transform Discovery

To find the right transform for a task:

1. By category: Refer to the categories above to narrow down the domain (CRUD, array, string, etc.)
2. By cheat sheet: Consult references/CHEAT_SHEET.md for complete transform index with arguments and examples
3. By schema: Read <transform-dir>/schema.json for detailed input/output contracts

Each transform directory contains:

• transform.jq - The jq implementation (5-40 lines)
• schema.json - Input/output contract with examples
• test.sh - Comprehensive test suite
• test-input-*.json - Test fixtures

Key Design Principles

1. Schema-first: Every transform has explicit input/output/args contract
2. Argument-based: All parameters via --arg/--argjson (no hardcoding)
3. Composable: Chain transforms via Unix pipes
4. Type-safe: Graceful handling of missing paths and wrong types
5. Self-contained: Each transform is isolated, no dependencies
6. Minimal: Fewer lines = fewer bugs, easier to understand

Common Patterns

Get value with default:

jq -f crud-get/transform.jq --arg path "config.timeout" --arg default "30"

Set nested value:

jq -f crud-set/transform.jq --arg path "user.profile.age" --arg value "30"

Filter and extract:

jq -f array-filter/transform.jq --arg array_field "users" --arg field "active" --arg value "true" | \
jq -f array-map/transform.jq --arg array_field "users" --arg field "email"

Validate required fields:

jq -f validate-required/transform.jq --arg fields "name,email,age"

Critical Implementation Notes

When creating new transforms or debugging issues:

Avoid common jq pitfalls:

• Use try-catch instead of // fallback for falsy value handling
• Use has() instead of != null for existence checks
• Check array types before using map() operations

Refer to references/IMPLEMENTATION_SPEC.md for complete implementation patterns, test coverage requirements, and detailed pitfall documentation.

References

This skill bundles comprehensive reference documentation (loaded only when needed):

• CHEAT_SHEET.md - Quick reference of all 60+ transforms with arguments and examples
• IMPLEMENTATION_SPEC.md - Complete implementation guide with patterns and pitfalls
• README.md - User-facing documentation and design principles
• USAGE_EXAMPLES.md - Real-world usage scenarios

Testing

Test transforms before use:

# Test specific transform
./crud-get/test.sh

# Test all transforms
./test-all.sh

Each transform has 8-17 test cases covering happy paths, falsy values, type safety, boundary conditions, and error cases.