APM Layer Skill

Layer Number: 11 Specification: Metadata Model Spec v0.7.0 Purpose: Defines observability using OpenTelemetry 1.0+, specifying traces, metrics, logs, and instrumentation.

Layer Overview

The APM Layer captures application performance monitoring:

• TRACES - Distributed tracing with spans
• METRICS - Performance and business metrics
• LOGS - Structured logging
• INSTRUMENTATION - Code instrumentation configuration
• RESOURCES - Telemetry resource attributes
• SLOs/SLAs - Service level objectives and agreements

This layer uses OpenTelemetry 1.0+ (industry standard for observability).

Central Entity: The Span (unit of work in trace) is the core modeling unit.

Entity Types

Core APM Entities (14 entities)

When to Use This Skill

Activate when the user:

• Mentions "APM", "observability", "monitoring", "telemetry"
• Wants to add tracing, metrics, or logging
• Asks about performance monitoring or SLOs
• Needs to instrument code or track business metrics
• Wants to link observability to application services

Cross-Layer Relationships

Outgoing (APM → Other Layers):

• instrumented-service → Application Layer (which service is being monitored?)
• business-metrics → Business Layer (business KPIs)

Incoming (Other Layers → APM):

• Application Layer → APM (services reference metrics)
• API Layer → APM (operations set SLA targets)
• Business Layer → APM (processes reference business metrics)
• Data Model Layer → APM (data quality metrics)
• Datastore Layer → APM (query performance metrics)

Observability Best Practices

1. Traces - Add spans for critical operations
2. Metrics - Track both technical and business metrics
3. Logs - Use structured logging (JSON)
4. Context - Propagate trace context across services
5. Sampling - Configure appropriate sampling rates
6. SLOs - Define service level objectives
7. Alerting - Set up alerts for critical metrics
8. Cardinality - Avoid high-cardinality attributes

Common Commands

# Add span
dr add apm span --name "Process Order"

# Add metric
dr add apm instrument-config --name "order_rate" --property type=counter

# Add log record
dr add apm log-record --name "Error Log"

# List APM elements
dr list apm span

# Validate APM layer
dr validate --layer apm

# Export APM configuration
dr export --layer apm --format json

Example: Order Processing Span

id: apm.span.process-order
name: "Process Order Span"
type: span
properties:
  spanKind: INTERNAL
  operationName: process_order
  attributes:
    - key: order.id
      type: string
    - key: order.total
      type: double
    - key: customer.id
      type: string
  events:
    - name: order_validated
      timestamp: relative
    - name: payment_processed
      timestamp: relative
    - name: inventory_reserved
      timestamp: relative
  instrumented-service: application.service.order-processing
  business-metrics:
    - apm.metric.orders-processed
    - apm.metric.order-processing-time
  sla-target-latency: 500ms
  sla-target-availability: 99.9%

Example: Business Metrics

id: apm.metric.order-rate
name: "Order Rate Metric"
type: instrument-config
properties:
  instrumentType: counter
  unit: orders
  description: "Number of orders processed"
  attributes:
    - key: order.type
      type: string
      values: [standard, express, subscription]
    - key: payment.method
      type: string
      values: [credit_card, paypal, bank_transfer]
  business-context:
    kpi: true
    dashboard: revenue-dashboard
    alerts:
      - threshold: 100
        condition: less-than
        severity: warning
        message: "Order rate below threshold"

Example: Data Quality Metrics

id: apm.data-quality-metrics.user-data
name: "User Data Quality Metrics"
type: data-quality-metrics
properties:
  entity: data_model.object-schema.user
  metrics:
    - name: completeness
      type: gauge
      description: "Percentage of non-null required fields"
      target: 95%
    - name: validity
      type: gauge
      description: "Percentage of records passing validation"
      target: 99%
    - name: uniqueness
      type: gauge
      description: "Percentage of unique email addresses"
      target: 100%
    - name: freshness
      type: histogram
      description: "Time since last update"
      target: 24h

Pitfalls to Avoid

• ❌ Not propagating trace context across services
• ❌ High-cardinality attributes (e.g., unique IDs in tags)
• ❌ Missing business metrics (only technical metrics)
• ❌ Not setting SLOs/SLAs
• ❌ Over-instrumentation (too many spans)
• ❌ Missing cross-layer links to instrumented services