High Availability Enhancement

EnerOS v0.41.0 introduces production-grade high availability capabilities, supporting multi-region deployment, WAL replication, disaster recovery (DR), CRDT state synchronization, and zero-downtime rolling upgrades. Combined with SLO/SLI error budgets and AIOps alert clustering, it provides 99.99% availability guarantee. The high availability capability is provided by the eneros-ha crate, working with eneros-multiregion and eneros-os.

High Availability Architecture

┌─────────────────────────────────────────────────────────────┐
│                       Global DNS / Load Balancing           │
│            Geographic proximity + health-based routing      │
└──────────────┬──────────────────────┬───────────────────────┘
               │                      │
┌──────────────▼──────────┐  ┌────────▼──────────────────────┐
│   Region A: cn-east-1   │  │   Region B: cn-south-1        │
│  ┌─────────────────┐    │  │  ┌─────────────────┐         │
│  │  Active Cluster │◀───┼──┼─▶│  Active Cluster │         │
│  │  3-node Raft    │    │  │  │  3-node Raft    │         │
│  └────────┬────────┘    │  │  └────────┬────────┘         │
│           │             │  │           │                  │
│      WAL Replication    │  │      WAL Replication         │
│   ┌───────┴───────┐     │  │   ┌───────┴───────┐          │
│   │  Replica 1/2  │     │  │   │  Replica 1/2  │          │
│   └───────────────┘     │  │   └───────────────┘          │
└─────────────────────────┘  └──────────────────────────────┘
               ▲                      ▲
               │   CRDT Async Merge   │
               └──────────────────────┘

Multi-Region Deployment

Supports both Active-Active and Active-Standby modes. In Active-Active mode, both regions handle traffic simultaneously, and write operations are merged through CRDT; in Active-Standby mode, the standby region synchronizes primary region data in real time, with second-level switchover on failure.

use eneros_ha::{Cluster, Region, RegionRole, Topology, Consensus};

let cluster = Cluster::new()
    .region(Region::new("cn-east-1")
        .role(RegionRole::Active)
        .nodes(3)
        .priority(100))
    .region(Region::new("cn-south-1")
        .role(RegionRole::Active)
        .nodes(3)
        .priority(90))
    .topology(Topology::ActiveActive)
    .consensus(Consensus::Raft)
    .replication_factor(3)
    .read_quorum(Quorum::Majority)
    .write_quorum(Quorum::Majority);

cluster.deploy().await?;

Deployment Mode Comparison

WAL Replication

All write operations are replicated to replicas through Write-Ahead Log, ensuring RPO ≈ 0. WAL uses a hybrid of synchronous and asynchronous replication: local replicas use synchronous replication for strong consistency, cross-region replicas use asynchronous replication to reduce latency.

use eneros_ha::wal::{WalConfig, Replication, Compression, KeyId};

let wal = WalConfig::new()
    .replication(Replication::hybrid()
        .sync_replicas(2)              // 2 local synchronous replicas
        .async_replicas(1)             // 1 cross-region asynchronous replica
        .ack_timeout(Duration::from_millis(500)))
    .flush_interval(Duration::from_millis(1))
    .segment_size(64 * 1024 * 1024)    // 64MB segment
    .compression(Compression::Zstd)
    .encryption(KeyId::from("wal-key"))
    .checksum(ChecksumAlgo::Crc32c);

kernel.enable_wal(wal).await?;

Replication Topology

Primary ──sync──▶ Replica 1 (local)
   │
   ├──sync──▶ Replica 2 (local)
   │
   └──async──▶ Replica 3 (cross-region, cn-south-1)

WAL Configuration Parameters

CRDT State Synchronization

For non-critical states (such as Agent memory, caches, counters), CRDT (Conflict-Free Replicated Data Type) is used to achieve eventual consistency, avoiding synchronous latency blocking write operations.

use eneros_ha::crdt::{CrdtStore, GCounter, ORSet, LwwMap};

let store = CrdtStore::new()
    .replication(Replication::all())
    .anti_entropy(Duration::from_secs(1));

// Grow-only counter (CRDT)
let counter = store.gcounter("agent.decisions");
counter.increment(1).await?;

// Set (CRDT)
let set = store.or_set("active.agents");
set.insert("dispatch_agent").await?;
set.remove("dispatch_agent").await?;

// Last-Write-Wins Map
let map = store.lww_map("agent.memory");
map.insert("dispatch_agent", memory_blob, now()).await?;

// Automatic cross-region merge, conflict-free

CRDT Type Matrix

Disaster Recovery (DR)

Automatically switches to standby region on regional failure, ensuring business continuity:

use eneros_ha::dr::{DrPlan, FailoverPolicy, Detection, DnsSwitch, HealthCheck};

let plan = DrPlan::new()
    .rto(Duration::from_secs(60))      // Recovery Time Objective
    .rpo(Duration::from_secs(0))       // Recovery Point Objective
    .failover_policy(FailoverPolicy::automatic(
        Detection::heartbeat(Duration::from_secs(10)),
        Confirmation::nodes(2),  // 2 nodes confirm failure
    ))
    .dns_switch(DnsSwitch::auto()
        .ttl(Duration::from_secs(30)))
    .health_check(HealthCheck::post_failover()
        .timeout(Duration::from_secs(60)))
    .notify(vec!["ops@grid.com", "soc@grid.com"]);

plan.activate().await?;

DR Drills

Regularly execute DR drills automatically to verify disaster recovery capabilities:

plan.drill()
    .simulate(Scenario::region_failure("cn-east-1"))
    .verify(SLO::availability(99.99))
    .verify(SLO::latency_p99(Duration::from_millis(100)))
    .schedule(Duration::from_secs(86400 * 30))  // Monthly
    .notify_on_failure(vec!["ops@grid.com"])
    .run().await?;

DR Key Metrics

Rolling Upgrade

Zero-downtime rolling upgrade, with real-time domain and general domain upgraded separately. The upgrade process supports canary release and automatic rollback:

use eneros_ha::upgrade::{RollingUpgrade, Strategy, HealthCheck};

let upgrade = RollingUpgrade::new("v0.47.0")
    .strategy(Strategy::canary(10)   // 10% canary
        .observe(Duration::from_secs(300))
        .promote_on(HealthCheck::all_healthy()))
    .batch_size(1)
    .batch_interval(Duration::from_secs(30))
    .health_check(Duration::from_secs(60))
    .rollback_on_failure(true)
    .rollback_threshold(0.05)         // Failure rate > 5% triggers rollback
    .realtime_domain_last(true);      // Real-time domain upgraded last

upgrade.execute().await?;

Upgrade Strategy Comparison

SLO / SLI Error Budget

use eneros_ha::slo::{Slo, Sli, ErrorBudget, Alert};

let slo = Slo::new()
    .availability(99.99)  // 4 nines
    .window(Duration::from_secs(86400 * 30))
    .sli(Sli::success_rate())
    .sli(Sli::latency_p99(Duration::from_millis(100)))
    .error_budget(ErrorBudget::consume_on_incident())
    .alert(Alert::on_budget_exhausted(0.2));  // Alert when 20% remaining

slo.activate().await?;

// View remaining error budget
let remaining = slo.error_budget_remaining().await?;
println!("Remaining error budget: {:.2}%", remaining * 100.0);

// View historical SLO achievement
let history = slo.history(Range::last(Duration::from_secs(86400 * 90))).await?;
for day in history {
    println!("{}: availability {:.4f}%, budget consumed {:.2}%",
        day.date, day.availability * 100.0, day.budget_consumed * 100.0);
}

SLO Levels

AIOps Alert Clustering

ML-based alert clustering and root cause analysis to avoid alert storms:

use eneros_ha::aiops::{AlertCorrelator, RootCauseAnalysis, CorrelationAlgorithm};

let correlator = AlertCorrelator::new()
    .algorithm(CorrelationAlgorithm::dbscan()
        .eps(Duration::from_secs(60))
        .min_samples(3))
    .window(Duration::from_secs(60))
    .dedup(true)
    .on_cluster(|cluster| async move {
        println!("Alert cluster: {} alerts", cluster.alerts.len());
        let rca = RootCauseAnalysis::analyze(&cluster).await?;
        println!("Root cause: {} (confidence {:.0}%)",
            rca.root_cause, rca.confidence * 100.0);
        notify_oncall(&rca).await?;
        Ok(())
    });

correlator.start().await?;

RCA Algorithms

Performance Metrics

High Availability Capability Matrix

Relationship with Other Capabilities

• Operations Automation: Operations automation is the execution layer of high availability, see Operations Automation
• Real-Time Dual Execution Domain: Real-time domain failover requires second-level, see Real-Time Dual Execution Domain
• Multi-tenant Isolation: Tenant-level high availability configures SLO per tenant, see Multi-tenant and Isolation
• Zero Trust: Cross-region replication enforces mTLS encryption, see Zero Trust and Security Enhancement
• Time-Series Native: Time-series data cross-region replication ensures RPO=0

Related Documentation

• Operations Automation — Operations supports high availability
• Real-Time Dual Execution Domain — Real-time domain failover
• Multi-tenant and Isolation — Tenant-level high availability
• Zero Trust and Security Enhancement — Cross-region encrypted replication
• Agent Intelligence Advanced — AIOps alert clustering