Operations Automation

EnerOS has built-in complete operations automation capabilities, covering On-call scheduling, Runbook automatic execution, automatic autoscaling, backup recovery and other scenarios, reducing dependence on manual operations and improving the stability of power production environments. Operations automation is provided by the eneros-ops crate, deeply integrated with the alert system, health checks, and security gateway.

Automation Architecture

┌──────────────────────────────────────────────────────────┐
│                    Monitoring and Alert Sources           │
│  Prometheus / Health Probe / IDS / SLO Violation          │
└──────────────────────┬───────────────────────────────────┘
                       │
┌──────────────────────▼───────────────────────────────────┐
│                  Automation Decision Layer                │
│  Alert Router ─▶ Runbook Engine ─▶ Approval Gate         │
│                       │                                   │
│              ┌────────┴────────┐                          │
│              ▼                 ▼                          │
│      Action Executor     On-call Notifier                │
└──────────────────────┬───────────────────────────────────┘
                       │
┌──────────────────────▼───────────────────────────────────┐
│                  Execution and Rollback Layer             │
│  Service Restart / Scale Up-Down / Backup / Restore      │
│  All actions fall into WORM audit chain                  │
└──────────────────────────────────────────────────────────┘

On-call Management

Supports multi-team, multi-level scheduling and escalation policies:

use eneros_ops::oncall::{OncallSchedule, Rotation, EscalationPolicy, Day, Time};
use std::time::Duration;

let schedule = OncallSchedule::new()
    .rotation(Rotation::weekly()
        .members(vec!["alice", "bob", "carol"])
        .handoff(Day::Monday, Time::from_hms(9, 0, 0)))
    .escalation(EscalationPolicy::new()
        .level(1, Duration::from_secs(5 * 60))    // 5 min no response → L2
        .level(2, Duration::from_secs(15 * 60))   // 15 min → L3
        .level(3, Duration::from_secs(30 * 60)))  // 30 min → L4
    .timezone("Asia/Shanghai")
    .follow_the_sun(true)  // Follow-the-sun mode
    .override(OncallOverride::new()
        .user("dave")
        .start("2026-07-10T09:00:00+08:00")
        .end("2026-07-15T18:00:00+08:00"));

schedule.activate().await?;

Escalation Level Description

Runbook Automation

Transforms operations playbooks into executable Runbooks, automatically executed when alerts trigger. Runbooks support conditional branching, parallel steps, and manual approval nodes.

use eneros_ops::runbook::{Runbook, Step, Trigger, Condition, Action};

let runbook = Runbook::new("high-cpu-agent")
    .trigger(Trigger::alert("agent_cpu_90"))
    .step(Step::check(health_check, "Service health check"))
    .step(Step::collect_diagnostic()
        .snapshot(Snapshot::heap_profile())
        .snapshot(Snapshot::thread_dump())
        .snapshot(Snapshot::metrics_window(Duration::from_secs(300))))
    .step(Step::conditional(
        Condition::cpu_high_for(Duration::from_secs(60)),
        Step::sequence(vec![
            Step::restart_service("agent-runtime"),
            Step::wait(Duration::from_secs(30)),
            Step::verify(ServiceHealth::healthy()),
        ]),
    ))
    .step(Step::approval(
        "Emergency scaling requires supervisor approval",
        Duration::from_secs(300),
        Approver::role("ops_lead"),
    ))
    .step(Step::notify_oncall()
        .channel(NotifyChannel::DingTalk));

runbook.register(&ops).await?;

Runbook Library

Runbook Configuration Parameters

Automatic Autoscaling

Automatically adjusts resources based on multi-dimensional metrics, supporting both Predictive and Reactive strategies:

use eneros_ops::autoscale::{Autoscaler, Rule, Strategy};

let scaler = Autoscaler::new()
    .target("agent-runtime")
    .min_instances(2)
    .max_instances(10)
    .strategy(Strategy::reactive())
    .rule(Rule::cpu(70, 90))           // CPU 70% scale up, 90% emergency scale up
    .rule(Rule::memory(80, 95))
    .rule(Rule::queue_depth(1000, 5000))
    .rule(Rule::custom("inflight_requests", 500, 2000))
    .cooldown(Duration::from_secs(300))
    .scale_up_step(2)                   // Scale up 2 instances each time
    .scale_down_step(1)
    .grace_period(Duration::from_secs(60));

scaler.activate().await?;

Predictive Autoscaling

Scales up in advance based on historical load predictions:

use eneros_ops::autoscale::predictive::{PredictiveScaler, ForecastModel};

let predictive = PredictiveScaler::new()
    .model(ForecastModel::prophet()
        .history(Duration::from_secs(86400 * 30))  // 30 days history
        .horizon(Duration::from_secs(3600))        // 1 hour prediction
        .confidence(0.95))
    .lookahead(Duration::from_secs(600))           // 10 minutes ahead
    .min_confidence(0.80);

predictive.activate().await?;

Autoscaling Rule Parameters

Real-time Domain Does Not Participate in Automatic Autoscaling

Real-time domain resources are statically bound at deployment time, avoiding dynamic scheduling that would break determinism. Autoscaling only applies to the general domain. Real-time domain scaling requires manual approval before execution.

Backup and Recovery

Supports full, incremental, and continuous backups, all backups are encrypted and written to object storage:

use eneros_ops::backup::{BackupManager, BackupSpec, BackupTarget, KeyId, Verification};

let backup = BackupManager::new()
    .schedule("daily-full", BackupSpec::full()
        .target(BackupTarget::S3("s3://eneros-backup"))
        .encryption(KeyId::from("backup-key"))
        .compression(Compression::Zstd { level: 9 })
        .retention(Duration::from_secs(86400 * 30))
        .verify(Verification::checksum()))
    .schedule("5min-incremental", BackupSpec::incremental()
        .target(BackupTarget::S3("s3://eneros-backup"))
        .retention(Duration::from_secs(86400 * 7)))
    .schedule("continuous-wal", BackupSpec::continuous_wal()
        .target(BackupTarget::S3("s3://eneros-backup/wal"))
        .retention(Duration::from_secs(86400)));

backup.start().await?;

Recovery Operation

use eneros_ops::backup::{Restore, RestoreTarget, RestorePoint};

let restore = Restore::new()
    .from(RestorePoint::latest())
    .to(RestoreTarget::NewCluster("staging-restore"))
    .include(vec!["topology", "timeseries", "agents"])
    .verify(Verification::data_integrity())
    .verify(Verification::service_health());

restore.execute().await?;

Recovery Drills

Regularly execute recovery drills automatically to verify backup validity:

backup.drill()
    .restore_to("staging")
    .verify(Verification::data_integrity())
    .verify(Verification::service_health())
    .verify(Verification::slo_met(99.9))
    .schedule(Duration::from_secs(86400 * 7))  // Weekly
    .notify_on_failure(vec!["ops@grid.com"])
    .run().await?;

Backup Strategy Comparison

Health Check and Self-Healing

Multi-dimensional health probing and automatic self-healing:

use eneros_ops::health::{HealthMonitor, Probe, ProbeType};

let monitor = HealthMonitor::new()
    .probe(Probe::http("agent-runtime", "/health", 200)
        .interval(Duration::from_secs(10))
        .timeout(Duration::from_secs(2))
        .failure_threshold(3))
    .probe(Probe::tcp("realtime-domain", 6080)
        .interval(Duration::from_secs(5)))
    .probe(Probe::custom("tsdb", check_tsdb_health)
        .interval(Duration::from_secs(30)))
    .probe(Probe::grpc("kernel", "/grpc.health.v1.Health/Check")
        .interval(Duration::from_secs(10)))
    .on_unhealthy(|service, status| async move {
        match status.failures {
            1..=2 => SelfHealing::restart(service).await,
            3..=5 => SelfHealing::replace(service).await,
            _ => SelfHealing::escalate(service).await,
        }
    });

monitor.start().await?;

Self-Healing Action Matrix

Performance Metrics

Notification Channels

Relationship with Other Capabilities

• High Availability Enhancement: Operations automation is the execution layer of HA guarantees, see High Availability Enhancement
• Multi-tenant Isolation: Operations actions isolated by tenant, see Multi-tenant and Isolation
• Safety Guard: Operations commands also pass through safety gateway, see Safety Guard
• Zero Trust: All operations actions fall into audit chain, see Zero Trust and Security Enhancement
• AIOps: Alert clustering and root cause analysis drive Runbook triggers

Related Documentation

• High Availability Enhancement — Operations supports high availability
• Multi-tenant and Isolation — Tenant-level operations
• Safety Guard — Operations commands also pass through safety gateway
• Zero Trust and Security Enhancement — Operations action audit
• Agent Intelligence Advanced — AIOps intelligent operations