Instance Actor Management

Overview

Instance actors are detached Duroxide orchestrations that run continuously to monitor PostgreSQL instance health. They can become orphaned or stale.

Actor Lifecycle

1. Created: When instance is created, create_instance orchestration spawns an actor
2. Running: Actor loops forever: check health → update CMS → timer → continue-as-new
3. Cancelled: When instance is deleted, actor is cancelled via cancel_instance()

Detecting Problems

Stale Actor (running but not working)

• last_health_check > 5 minutes old
• Actor may be stuck, timer broken, or worker not processing

Missing Actor

• instance_actor_orchestration_id is NULL
• Or orchestration doesn't exist in duroxide state

Zombie Actor (in DB but not processing)

• Status shows "Running" but no health updates
• Often caused by server crash or DB migration

API Endpoints

# Start new actor
curl -X POST /api/instances/:name/actor/start

# Restart actor (cancel + start new)
curl -X POST /api/instances/:name/actor/restart

# Cancel actor
curl -X POST /api/instances/:name/actor/cancel

Duroxide Client Usage

// Cancel an actor
client.cancel_instance(&actor_id, "User requested cancellation").await?;

// Check if actor exists
match client.get_instance_info(&actor_id).await {
    Ok(info) => println!("Status: {}", info.status),
    Err(_) => println!("Actor not found"),
}

// Start new actor (detached)
client.start_orchestration(
    &new_actor_id,
    orchestrations::INSTANCE_ACTOR,
    serde_json::to_string(&input)?,
).await?;

Recovery Procedures

Restart All Stale Actors

-- Find instances with stale health
SELECT user_name, k8s_name, last_health_check, instance_actor_orchestration_id
FROM toygres_cms.instances
WHERE state = 'running'
  AND (last_health_check IS NULL OR last_health_check < NOW() - INTERVAL '5 minutes');

Then use the UI or API to restart each actor.

After Database Migration

If duroxide state wasn't migrated, all actors are orphaned:

1. List all running instances
2. For each, call /api/instances/:name/actor/start