HA Cluster Deployment

This tutorial demonstrates how to deploy a 3-node EnerOS high availability cluster that meets the dispatch master station disaster recovery requirements of RTO < 5s and RPO = 0.

HA Architecture Overview

EnerOS’s HA capability is provided by the eneros-os::ha module, which contains the following submodules:

Failover state machine:

            ┌──────────┐
            │ Standby  │ ←─────────────┐
            └────┬─────┘               │
                 │ take_over()         │ fail_back()
            ┌────▼─────┐          ┌────┴──────┐
            │ TakingOver│ ───────►│  Active   │
            └──────────┘          └────┬──────┘
                                       │ primary_down()
                                  ┌────▼──────┐
                                  │ FailingBack│
                                  └────┬──────┘
                                       │ error
                                  ┌────▼──────┐
                                  │   Failed  │
                                  └───────────┘

Prerequisites

• 3 physical or virtual machines with identical configuration (16C / 64G / NVMe)
• Shared storage or distributed block storage (for Raft WAL persistence)
• Installation and Build completed
• Internal CA configured, with inter-node mTLS certificates issued
• NTP configured (inter-node clock skew < 50ms, otherwise heartbeat misdetection)
• SELinux / firewalld disabled, or UDP 5400, TCP 5401, TCP 7878 ports opened

Step 1: Plan the Topology

The database uses Raft three-replica replication to guarantee strong consistency. The Witness node carries no data; it only participates in Quorum voting to break even-node deadlocks.

Quorum arbitration algorithm:

• total_members: number of non-Witness members (3 in this example)
• alive_count: number of Alive non-Witness members
• witness_count: number of Witness nodes (1 in this example)
• Has majority: alive_count + witness_count > (total_members + witness_count) / 2
• Leader: the surviving non-Witness member with the highest priority

In this example:

• Normal state: alive=3, witness=1, total=3 → 3+1 > 2 ✓ Has Quorum, Leader = node-1
• node-1 down: alive=2, witness=1, total=3 → 2+1 > 2 ✓ Has Quorum, Leader = node-2 (next highest priority)
• node-1 + node-2 down: alive=1, witness=1, total=3 → 1+1 > 2 ✗ No Quorum, service degraded

Step 2: Configure the Cluster

Complete configuration example for /etc/eneros/ha.toml:

# /etc/eneros/ha.toml
# EnerOS high availability configuration

# Required fields
node_id = "node-1"
role = "primary"

# Heartbeat parameters (milliseconds)
heartbeat_interval_ms = 100      # Heartbeat send interval
heartbeat_suspect_ms = 100       # Suspect threshold (>interval triggers suspicion)
heartbeat_dead_ms = 300          # Dead threshold (>suspect triggers offline)

# UDP multicast address (must be in the 224.0.0.0/4 range)
multicast_addr = "239.0.0.1"
heartbeat_port = 5400
sync_port = 5401
multicast_ttl = 32

# Node priority (higher number means higher priority, used during Leader election)
priority = 200

# Dual-NIC redundancy
interfaces = ["eth0", "eth1"]

# Fencing strategy (must be non-None in production)
fencing_strategy = "stonith"     # Options: none / stonith / disk / network

# Production flag (when true, enforces fencing_strategy != none)
is_production = true

# HMAC authentication key (protects heartbeat/sync messages, must be identical on all nodes)
auth_key = "ha-shared-secret-change-me"

# Sync scope
[sync_scope]
sync_scada = true                # SCADA real-time data
sync_agent_state = true          # Agent state
sync_command_history = true      # Command history
sync_config = true               # Configuration files

# Failover configuration
[failover]
vip = "10.0.0.10/24"             # Virtual IP (CIDR notation)
vip_interface = "eth0"           # NIC bound to the VIP
cleanup_arp = true               # Send gratuitous ARP on switchover
takeover_timeout_ms = 3000       # Takeover timeout (3 seconds)
recovery_policy = "auto_prefer_primary"  # or "manual"
cross_region_takeover_timeout_ms = 60000  # Cross-region takeover timeout (60 seconds)

# Cluster configuration (multi-node Quorum)
[cluster]
quorum_policy = "majority"       # or "manual"

[[cluster.members]]
node_id = "node-1"
role = "primary"
priority = 200
region_id = "region-a"

[[cluster.members]]
node_id = "node-2"
role = "secondary"
priority = 150
region_id = "region-a"

[[cluster.members]]
node_id = "node-3"
role = "secondary"
priority = 100
region_id = "region-b"

# Witness node (carries no data, only votes)
cluster.witness = ["node-w"]

# Region definitions (v0.41.0 multi-region support)
[[cluster.regions]]
id = "region-a"
datacenter = "dc-beijing"
priority = 200
endpoints = ["10.0.0.11:7878", "10.0.0.12:7878"]

[[cluster.regions]]
id = "region-b"
datacenter = "dc-shanghai"
priority = 100
endpoints = ["10.0.0.13:7878"]

# Disaster recovery drill configuration
[drill]
enabled = true
schedule = "weekly"              # daily / weekly / monthly
auto_rollback = true

HaConfig field descriptions:

Configuration validation rules (HaConfig::validate):

1. heartbeat_suspect_ms < heartbeat_dead_ms (suspect must be less than dead)
2. heartbeat_interval_ms > 0 (interval must be greater than 0)
3. heartbeat_suspect_ms >= heartbeat_interval_ms (suspect must be ≥ interval)
4. multicast_addr is in the 224.0.0.0/4 range (first byte 224-239)
5. heartbeat_port != sync_port (ports cannot conflict)
6. node_id is non-empty
7. In production (is_production == true), fencing_strategy != None
8. failover.takeover_timeout_ms > 0
9. failover.vip (if non-empty) must be a valid IPv4/IPv6 address
10. cluster.members is non-empty
11. Node IDs in cluster.witness must not appear in cluster.members
12. node_id must be in cluster.members
13. drill.scenarios is non-empty (if drill.enabled == true)

Step 3: Start the Services

Start in the order node-1 → node-2 → node-3 → node-w:

# node-1 (Primary)
ssh node-1
sudo systemctl start eneros-os
journalctl -u eneros-os -f | grep -E "ha|cluster|heartbeat"

# node-2 (Secondary)
ssh node-2
sudo systemctl start eneros-os

# node-3 (Secondary)
ssh node-3
sudo systemctl start eneros-os

# node-w (Witness, lightweight process carrying no data)
ssh node-w
sudo systemctl start eneros-witness

The cluster automatically completes Leader election and interconnects via mTLS. Example startup logs:

2026-07-06T10:00:00.123Z INFO  eneros_os::ha::heartbeat  node-1 started heartbeat on 239.0.0.1:5400
2026-07-06T10:00:00.234Z INFO  eneros_os::ha::sync        sync server listening on 0.0.0.0:5401
2026-07-06T10:00:00.345Z INFO  eneros_os::ha::cluster     cluster initialized: members=3, witness=1
2026-07-06T10:00:00.456Z INFO  eneros_os::ha::cluster     quorum acquired: alive=3, leader=node-1
2026-07-06T10:00:00.567Z INFO  eneros_os::ha::failover    state=Active, vip=10.0.0.10, readonly=false
2026-07-06T10:00:00.678Z INFO  eneros_os::ha::fencing     fencing_strategy=Stonith, cooldown=30s

Step 4: Load Configuration and Start the HA Engine

Programmatically start the full HA stack:

use std::sync::Arc;
use eneros_os::ha::{
    BatchConfig, ClusterConfig, FailoverConfig, FailoverEngine, FencingManager,
    FencingStrategy, HaConfig, HeartbeatManager, NodeRole, RecoveryPolicy,
    SharedStore, SyncManager, SyncScope, ClusterManager,
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Load the HA configuration
    let config = HaConfig::load("/etc/eneros/ha.toml")?;
    println!("loaded ha config: node_id={}, role={:?}", config.node_id, config.role);

    // 2. Initialize the shared state store (with WAL persistence)
    let store = Arc::new(SharedStore::new(
        config.node_id.clone(),
        config.role,
        "/var/lib/eneros/ha-store",  // WAL file directory
    )?);
    store.load_from_disk()?;  // Restore state after restart
    println!("shared store loaded: entries={}", store.len());

    // 3. Start the heartbeat manager
    let heartbeat = Arc::new(HeartbeatManager::new(config.clone())?);
    heartbeat.start().await?;
    println!("heartbeat started: interval={}ms", config.heartbeat_interval_ms);

    // 4. Start the sync manager
    let batch_config = BatchConfig {
        batch_size: 100,
        batch_timeout_ms: 10,
    };
    let sync = Arc::new(SyncManager::new(
        config.clone(),
        store.clone(),
        batch_config,
    )?);
    sync.start().await?;
    println!("sync started: port={}", config.sync_port);

    // 5. Start the Fencing manager
    let fencing = Arc::new(FencingManager::new(
        config.node_id.clone(),
        config.role,
        config.fencing_strategy,
    )?);
    println!("fencing ready: strategy={:?}", config.fencing_strategy);

    // 6. Start the cluster manager (multi-node Quorum)
    let cluster_config = config.cluster.clone()
        .expect("cluster config required for multi-node HA");
    let cluster = Arc::new(ClusterManager::new(
        cluster_config,
        config.node_id.clone(),
    ));
    println!("cluster manager initialized: members={}",
        cluster.config_read().members.len());

    // 7. Start the Failover engine
    let failover_config = config.failover.clone()
        .expect("failover config required");
    let failover = Arc::new(FailoverEngine::new(
        config.clone(),
        store.clone(),
        failover_config,
    ).with_sync_manager(sync.clone()));
    println!("failover engine: initial_state={:?}", failover.current_state());

    // 8. Register member-change callback
    cluster.register_member_callback(Arc::new(|event| {
        tracing::info!(
            "Member change: member={}, status={:?}, cluster_size={}",
            event.member_id, event.status, event.cluster_size
        );
    }));

    // 9. Main loop: monitor heartbeat timeouts and trigger failover
    let heartbeat_clone = heartbeat.clone();
    let failover_clone = failover.clone();
    let cluster_clone = cluster.clone();
    tokio::spawn(async move {
        let mut interval = tokio::time::interval(
            std::time::Duration::from_millis(50)
        );
        loop {
            interval.tick().await;
            // Check heartbeat timeouts
            let changes = heartbeat_clone.check_timeouts();
            for change in changes {
                tracing::warn!(
                    "Node state change: {} {:?} -> {:?}",
                    change.node_id, change.old_state, change.new_state
                );
                // Update cluster member state
                cluster_clone.update_member_state(
                    &change.node_id,
                    change.new_state,
                );
                // If Leader goes down, trigger failover
                if change.new_state == eneros_os::ha::NodeState::Dead
                    && change.node_id != config.node_id {
                    // Quorum decides whether to take over
                    let quorum = cluster_clone.evaluate_quorum();
                    if let eneros_os::ha::QuorumResult::Leader(new_leader) = quorum {
                        if new_leader == config.node_id {
                            tracing::info!("This node was elected as the new Leader, triggering takeover");
                            let _ = failover_clone.take_over().await;
                        }
                    }
                }
            }
        }
    });

    // Wait for exit signal
    tokio::signal::ctrl_c().await?;
    println!("shutting down...");
    Ok(())
}

Step 5: Configure Failover

Place a load balancer (HAProxy / Keepalived) in front of the Gateway, with health checks pointed at /healthz:

# /etc/haproxy/haproxy.cfg
frontend eneros_frontend
    bind 10.0.0.10:443 ssl crt /etc/eneros/certs/gateway.pem
    mode http
    default_backend eneros_gateway

backend eneros_gateway
    mode http
    option httpchk GET /healthz
    http-check expect status 200
    # Health check interval 1s, remove after 3 failures
    default-server inter 1s fall 3 rise 2
    # Primary node has higher weight; backup nodes only receive traffic when primary is unavailable
    server node-1 10.0.0.11:8080 check weight 100
    server node-2 10.0.0.12:8080 check weight 50 backup
    server node-3 10.0.0.13:8080 check weight 50 backup

Keepalived configuration (VIP dual-node hot standby):

# /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass ener0s_ha
    }
    virtual_ipaddress {
        10.0.0.10/24
    }
    track_script {
        chk_eneros
    }
}

vrrp_script chk_eneros {
    script "/usr/local/bin/check-eneros-health.sh"
    interval 2
    fall 2
    rise 2
}

check-eneros-health.sh health check script:

#!/bin/bash
# /usr/local/bin/check-eneros-health.sh
# Returns 0 for healthy, 1 for unhealthy

# 1. Check the HTTP health endpoint
if ! curl -sf http://127.0.0.1:8080/healthz > /dev/null; then
    exit 1
fi

# 2. Check whether FailoverEngine is Active
STATE=$(curl -sf http://127.0.0.1:8080/api/v1/ha/status | jq -r '.failover_state')
if [ "$STATE" != "active" ]; then
    exit 1
fi

# 3. Check whether Quorum is held
QUORUM=$(curl -sf http://127.0.0.1:8080/api/v1/ha/quorum | jq -r '.has_quorum')
if [ "$QUORUM" != "true" ]; then
    exit 1
fi

exit 0

Step 6: Disaster Recovery Drills

DrillScheduler supports periodic automated drills to verify the reliability of the failover flow.

use eneros_os::ha::{
    DrillConfig, DrillScenario, DrillSchedule, DrillScheduler,
    FailoverEngine,
};
use std::sync::Arc;

fn setup_drill_scheduler(
    failover: Arc<FailoverEngine>,
) -> anyhow::Result<Arc<DrillScheduler>> {
    let config = DrillConfig {
        enabled: true,
        schedule: DrillSchedule::Weekly,  // Drill once per week
        auto_rollback: true,
        scenarios: vec![
            DrillScenario::PrimaryDown,
            DrillScenario::NetworkPartition,
            DrillScenario::DiskFailure,
            // v0.41.0 cross-region scenarios
            DrillScenario::CrossRegionFailover {
                from_region: "region-a".to_string(),
                to_region: "region-b".to_string(),
            },
            DrillScenario::DataCenterLoss {
                region: "region-a".to_string(),
            },
        ],
    };

    let scheduler = Arc::new(DrillScheduler::new(config, failover));
    scheduler.start()?;  // Start the background scheduling task
    println!("drill scheduler started: schedule={:?}", DrillSchedule::Weekly);

    Ok(scheduler)
}

Drill scenario descriptions:

Manually trigger a drill:

# Run the PrimaryDown drill immediately
eneros-cli ha drill run --scenario primary_down --auto-rollback

# View drill history
eneros-cli ha drill history --limit 10

# View the next scheduled time
eneros-cli ha drill next

Drill logs are recorded in /var/log/eneros/drill.log (JSON Lines format), with up to 50 historical records retained.

Step 7: Validate Disaster Recovery

# 1. Simulate Leader failure
ssh node-1 sudo systemctl stop eneros-os

# 2. Observe cluster state changes (a new Leader is elected within 5 seconds)
eneros-cli cluster status --watch

# 3. Verify business continuity (SCADA data not interrupted)
eneros-cli scada query --topic /topics/voltage --last 1m

# 4. Verify RPO = 0 (no data loss)
eneros-cli ha failover history --last 1 | jq '.records[0] | {duration_ms, result, reason}'

Expected output:

=== Cluster Status ===
node-1: DEAD    (last_seen: 2026-07-06T10:05:00Z)
node-2: ALIVE   (role: primary, priority: 150)  ← New Leader
node-3: ALIVE   (role: secondary, priority: 100)
node-w: ALIVE   (role: witness)

Quorum: ACQUIRED (alive=2, witness=1, total=3)
Failover: ACTIVE (state=active, vip=10.0.0.10, readonly=false)
Last Failover: 2026-07-06T10:05:03.123Z (duration=2.8s, reason="primary_down")

Fault injection test matrix:

Step 8: Monitoring and Alerting

Connect Prometheus to scrape /metrics; key metrics are as follows:

Prometheus scrape configuration:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: eneros-ha
    scrape_interval: 5s
    scrape_timeout: 3s
    static_configs:
      - targets:
          - 10.0.0.11:8080
          - 10.0.0.12:8080
          - 10.0.0.13:8080
    metrics_path: /metrics

Grafana alert rule examples:

groups:
  - name: eneros-ha
    rules:
      - alert: EnerOSNoQuorum
        expr: eneros_ha_quorum_status == 0
        for: 10s
        labels:
          severity: critical
        annotations:
          summary: "EnerOS cluster has no Quorum ({{ $labels.node_id }})"
          description: "Node {{ $labels.node_id }} lost Quorum; service may be degraded"

      - alert: EnerOSLeaderFlapping
        expr: rate(eneros_cluster_leader_changes_total[1h]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Frequent Leader changes"
          description: "Leader changed {{ $value }} times within 1 hour"

      - alert: EnerOSSyncLagHigh
        expr: eneros_ha_sync_lag_ms > 500
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "State sync latency too high"
          description: "Node {{ $labels.node_id }} sync latency {{ $value }}ms"

      - alert: EnerOSFencingTriggered
        expr: increase(eneros_ha_fencing_triggered_total[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "Fencing triggered"
          description: "Split-brain protection has been activated; investigate the network immediately"

Step 9: Split-Brain Protection

Fencing strategy comparison:

Split-brain detection logic (FencingManager::detect_split_brain):

1. dead_nodes is empty → NoSplitBrain
2. The local node is in the dead node list → the local node should be fenced
3. Has quorum nodes: more than half reachable → the local node has quorum, the peer should be fenced; otherwise the local node should be fenced
4. No quorum nodes (two-node): Primary → the peer should be fenced; Secondary → conservative policy, the local node should be fenced

Security hardening features:

• Reject self-fencing: target_node == self.node_id → returns InvalidTarget
• 30-second cooldown: repeated requests for the same target within the cooldown return Skipped
• Audit log: every fencing record is appended to /var/log/eneros/fencing.log (JSON Lines)
• Quorum validation (v0.29.0): Quorum must be held before executing fencing, otherwise returns NoQuorum

use eneros_os::ha::{
    FencingManager, FencingStrategy, QuorumState,
};

async fn demonstrate_fencing() -> anyhow::Result<()> {
    let mut fencing = FencingManager::new(
        "node-1".to_string(),
        eneros_os::ha::NodeRole::Primary,
        FencingStrategy::Stonith,
    )?;

    // Update Quorum state (must be called first; defaults to single-node holding Quorum)
    fencing.update_quorum_state(QuorumState {
        has_quorum: true,
        alive_count: 3,
        total_count: 4,  // Including witness
    });

    // Detect split-brain
    let result = fencing.detect_split_brain(
        &["node-2"],               // dead_nodes
        &["node-w"],               // quorum_nodes
        std::time::Duration::from_millis(300),
    )?;
    println!("split-brain result: {:?}", result);

    // Execute fencing (only when the local node has Quorum)
    match fencing.fence("node-2", "primary_down_no_response").await {
        Ok(record) => println!("fencing success: {:?}", record.result),
        Err(e) => println!("fencing failed: {}", e),
    }

    Ok(())
}

Step 10: State Synchronization and Conflict Resolution

SharedStore provides an application-level replication engine; writes on the primary trigger replication callbacks, and the backup receives data via replicate.

use eneros_os::ha::{
    ConflictResolution, SharedStore, StorageEntry, StorageQuota,
};
use std::sync::Arc;

fn setup_shared_store() -> anyhow::Result<Arc<SharedStore>> {
    let store = Arc::new(SharedStore::new(
        "node-1".to_string(),
        eneros_os::ha::NodeRole::Primary,
        "/var/lib/eneros/ha-store",
    )?);

    // Set storage quota
    store.set_quota(StorageQuota {
        max_entries: 1_000_000,
        max_bytes: 1_073_741_824,  // 1 GB
    })?;

    // Set conflict resolution policy
    store.set_conflict_resolution(ConflictResolution::TimestampWins);

    // Register replication callback (triggered when the primary writes)
    let store_for_cb = store.clone();
    store.set_replicate_callback(Box::new(move |entry: StorageEntry| {
        // Serialize and send to the backup via SyncManager
        let store = store_for_cb.clone();
        tokio::spawn(async move {
            if let Err(e) = store.send_to_replica(&entry).await {
                tracing::error!("replicate failed: key={}, err={}", entry.key, e);
            }
        });
    }))?;

    Ok(store)
}

// Primary write example
fn write_scada_snapshot(store: &SharedStore, bus_id: u32, voltage: f32) -> anyhow::Result<()> {
    let entry = StorageEntry {
        key: format!("scada/voltage/{}", bus_id),
        value: serde_json::json!({
            "bus_id": bus_id,
            "voltage_pu": voltage,
            "timestamp": chrono::Utc::now().timestamp_millis(),
        }),
        timestamp: chrono::Utc::now().timestamp_millis(),
        node_id: store.node_id().to_string(),
        version: store.next_version(),
    };
    store.put(entry)?;  // Triggers replication callback
    Ok(())
}

// Backup receives replicated data
fn receive_replica_data(store: &SharedStore, entry: StorageEntry) -> anyhow::Result<()> {
    match store.replicate(entry.clone()) {
        Ok(()) => tracing::debug!("replicated: key={}", entry.key),
        Err(e) => tracing::warn!("replicate conflict: key={}, err={}", entry.key, e),
    }
    Ok(())
}

Conflict resolution strategy comparison:

Service degradation mode:

// Backup read-only protection (auto-called during FailoverEngine switchover)
fn enable_readonly_mode(store: &SharedStore) {
    store.set_readonly(true);
    tracing::warn!("Shared store entered read-only mode (backup node)");
}

// Lift read-only on promotion to primary
fn disable_readonly_mode(store: &SharedStore) {
    store.set_readonly(false);
    tracing::info!("Shared store read-only lifted (promoted to primary)");
}

Step 11: Persistence and Recovery

SharedStore supports snapshot + WAL persistence; ha-daemon automatically recovers after restart:

use eneros_os::ha::SharedStore;

fn demonstrate_persistence() -> anyhow::Result<()> {
    // Node running: periodic snapshots
    let store = SharedStore::new(
        "node-1".to_string(),
        eneros_os::ha::NodeRole::Primary,
        "/var/lib/eneros/ha-store",
    )?;

    // Auto-triggers a snapshot when WAL record count reaches 1000
    // Can also be triggered manually
    store.snapshot()?;
    println!("snapshot created: entries={}", store.len());

    // Recover after restart
    let restored = SharedStore::new(
        "node-1".to_string(),
        eneros_os::ha::NodeRole::Primary,
        "/var/lib/eneros/ha-store",
    )?;
    restored.load_from_disk()?;
    println!("restored from disk: entries={}", restored.len());

    Ok(())
}

WAL file structure:

/var/lib/eneros/ha-store/
├── snapshot.bin          # Latest snapshot
├── wal.log               # WAL log (append-only)
├── wal.index             # WAL index
└── manifest.json         # Metadata (version, last_snapshot_at, ...)

Recovery flow:

1. Read manifest.json for the last snapshot info
2. Load snapshot.bin into memory
3. Replay records after the snapshot point in wal.log
4. If WAL is corrupted, fall back to the previous snapshot

Validation

Validate whether the HA cluster meets the standard against the following checklist:

RTO/RPO validation script:

#!/bin/bash
# /usr/local/bin/verify-ha-rto-rpo.sh
# Continuously write test data, record timestamps, and validate after simulating a fault

set -e
VIP="10.0.0.10"
TEST_FILE="/tmp/ha-rto-test-$(date +%s).log"

echo "Starting RTO/RPO validation..."

# 1. Continuously write test data (one record every 10ms, for 60s)
for i in $(seq 1 6000); do
    TIMESTAMP=$(date +%s%3N)
    curl -s -X POST "https://${VIP}:443/api/v1/test/echo" \
        -H "Content-Type: application/json" \
        -d "{\"seq\": $i, \"ts\": $TIMESTAMP}" >> "$TEST_FILE" 2>/dev/null || \
        echo "FAIL,$i,$TIMESTAMP" >> "$TEST_FILE"
    sleep 0.01
done &

WRITER_PID=$!

# 2. Wait 10s then inject the fault
sleep 10
echo "Injecting fault: stopping node-1..."
ssh node-1 sudo systemctl stop eneros-os
FAULT_TIME=$(date +%s%3N)
echo "Fault time: $FAULT_TIME"

# 3. Wait for the writer to finish
wait $WRITER_PID

# 4. Validate: find the first successful record after the first FAIL
RECOVERY_TIME=$(grep -v "^FAIL" "$TEST_FILE" | awk -F',' '$2 > '"$FAULT_TIME"' {print $3; exit}')
if [ -z "$RECOVERY_TIME" ]; then
    echo "FAIL: not recovered"
    exit 1
fi

RTO_MS=$((RECOVERY_TIME - FAULT_TIME))
echo "RTO: ${RTO_MS}ms"

# 5. Validate RPO (should have 0 data loss)
TOTAL_SUCCESS=$(grep -v "^FAIL" "$TEST_FILE" | wc -l)
TOTAL_FAIL=$(grep "^FAIL" "$TEST_FILE" | wc -l)
echo "Successful writes: $TOTAL_SUCCESS, failures: $TOTAL_FAIL"
echo "RPO: $TOTAL_FAIL records lost"

# 6. Result judgment
if [ "$RTO_MS" -lt 5000 ] && [ "$TOTAL_FAIL" -eq 0 ]; then
    echo "PASS: RTO < 5s, RPO = 0"
    exit 0
else
    echo "FAIL: not meeting the standard"
    exit 1
fi

Debugging and Troubleshooting

View HA internal state:

# View cluster status
eneros-cli cluster status --json | jq

# View Failover status
eneros-cli ha status --json | jq

# View heartbeat latency
eneros-cli ha heartbeat latency

# View sync statistics
eneros-cli ha sync stats

# View shared store entry count
eneros-cli ha store stats

# View Fencing history
eneros-cli ha fencing history --limit 10

# View disaster recovery drill history
eneros-cli ha drill history --limit 10

# View Failover history
eneros-cli ha failover history --limit 10

HA log files:

Performance Tuning

Next Steps

• HA Enhancement — HA capability overview
• Multi-Tenant Deployment — overlay multi-tenancy on an HA cluster
• Plugin Development — HA-aware plugin development