EnerOS v0.16.0

Release Date: March 9, 2025 Codename: Raft Git Tag: v0.16.0 Support Status: Stable Total Crates: 42 (4 new) Test Cases: 5820+ (580 new)

Version Overview

EnerOS v0.16.0 “Raft” is the core release for EnerOS in the “High Availability” direction. The primary goal is to introduce a distributed consistency layer based on the Raft consensus algorithm, enabling EnerOS clusters to automatically switch Leaders, replicate logs, and maintain data consistency during node failures, achieving 99.99%+ availability to meet the stringent continuous operation requirements of power dispatch systems.

Power dispatch systems are typical “7×24 uninterrupted” systems—any downtime may create grid operational risks. Traditional solutions rely on active-standby hot standby, but issues such as reliance on external arbitration for failover, data synchronization windows, and lagging fault detection are prominent. The Raft algorithm solves distributed consistency problems at the algorithmic level through three mechanisms: Leader election, log replication, and majority commit, and is easier to understand and implement than Paxos. v0.16.0 deeply integrates Raft into the EnerOS kernel, providing strong consistency guarantees for critical data such as topology state, twin snapshots, tenant configurations, and security policies.

This version introduces four new crates: eneros-raft (Raft core implementation), eneros-raft-log (log replication), eneros-raft-election (Leader election), and eneros-raft-failover (failover). The design philosophy is “majority is truth”—any write must be confirmed by a majority of cluster nodes before commit, ensuring zero data loss and zero service interruption during minority node failures.

Key Metrics

New Features

1. Raft Consensus Algorithm

Introduced the eneros-raft crate, implementing the complete Raft consensus algorithm, including Leader election, log replication, safety guarantees, and membership changes. The implementation strictly follows the Raft paper (Ongaro & Ousterhout, 2014) and is validated through Jepsen-style distributed testing.

Cluster Initialization

use eneros_raft::{RaftNode, RaftConfig, NodeId};

// Create a 5-node Raft cluster
let nodes = vec![
    NodeId::from("node-1"),
    NodeId::from("node-2"),
    NodeId::from("node-3"),
    NodeId::from("node-4"),
    NodeId::from("node-5"),
];

let mut raft = RaftNode::new(NodeId::from("node-1"))
    .config(RaftConfig {
        peers: nodes.clone(),
        election_timeout: Duration::milliseconds(150..300),
        heartbeat_interval: Duration::milliseconds(50),
        commit_quorum: Quorum::Majority,
        log_storage: LogStorage::persistent("/var/eneros/raft"),
        snapshot_interval: Duration::minutes(10),
        snapshot_threshold: 10000,
    })?;

// Start Raft
raft.start().await?;

Raft Node States

Write Flow

// Client write (automatically routed to Leader)
let proposal = RaftProposal::new("update-topology")
    .payload(Command::UpdateBus {
        id: 1,
        voltage: 1.025,
    });

match raft.submit(proposal).await {
    Ok(commit) => {
        // Committed on majority nodes
        println!("Commit index: {}", commit.index);
        println!("Commit time: {:?}", commit.timestamp);
    }
    Err(RaftError::NotLeader(leader_hint)) => {
        // Redirect to Leader
        let leader = leader_hint.unwrap();
        redirect_to(leader, proposal).await?;
    }
}

2. Leader Election

Introduced the eneros-raft-election crate, implementing the Raft Leader election algorithm. When a Leader fails or network partition occurs, remaining nodes spontaneously initiate elections after the election timeout; the node receiving majority votes becomes the new Leader, with the entire election process completing within 1 second.

Election Flow

use eneros_raft_election::{Election, ElectionResult};

// Listen for election events
raft.on_election(|result| {
    match result {
        ElectionResult::Elected { term, votes } => {
            log::info!("New Leader elected: term={}, votes={}/{}",
                term, votes.for, votes.total);
        }
        ElectionResult::Lost { term, winner } => {
            log::info!("Election lost: term={}, winner={}", term, winner);
        }
        ElectionResult::Split { term } => {
            log::warn!("Election split: term={}", term);
        }
    }
});

Election Parameter Tuning

Priority Election

// Configure different priorities for different nodes, preferentially electing high-config nodes as Leader
let raft = RaftNode::new(NodeId::from("node-1"))
    .config(RaftConfig {
        peers: nodes,
        election_priority: Priority::High,  // High priority
        // ...
    })?;

// node-2: priority Medium
// node-3: priority Medium
// node-4: priority Low
// node-5: priority Low

3. Log Replication

Introduced the eneros-raft-log crate, implementing the Raft log replication mechanism. The Leader encapsulates all write operations as log entries, replicating them in parallel to all Followers, committing after majority confirmation. Logs are stored using LSM trees, supporting fast lookup and compaction.

Log Structure

use eneros_raft_log::{LogEntry, LogIndex, Term};

// Log entry structure
let entry = LogEntry {
    index: LogIndex(1024),
    term: Term(5),
    entry_type: EntryType::Normal,
    data: Command::UpdateBus { id: 1, voltage: 1.025 }.encode()?,
    timestamp: now(),
};

// Append log
let index = raft.log().append(entry).await?;

// Replicate to Followers
raft.log().replicate(index, &followers).await?;

// Commit log
raft.log().commit(index).await?;

Log Compaction and Snapshots

use eneros_raft_log::snapshot;

// Automatically snapshot and compact when logs exceed threshold
let snapshot = snapshot::create(&raft)
    .last_included_index(5000)
    .last_included_term(4)
    .state(topology.current_state()?)
    .build()?;

raft.log().install_snapshot(snapshot).await?;

// Log compaction effect
// Before compaction: 10000 log entries, size 120MB
// After compaction: 1 snapshot + 5000 log entries, size 8MB + 6MB = 14MB

Replication Performance Metrics

4. Automatic Failover

Introduced the eneros-raft-failover crate, implementing automatic detection and failover for node failures. When a Leader node fails, the cluster automatically elects a new Leader after the election timeout, client requests are automatically redirected, and the entire process is transparent to upper-layer applications.

Failure Detection

use eneros_raft_failover::{FailureDetector, DetectorConfig};

let detector = FailureDetector::new(DetectorConfig {
    heartbeat_timeout: Duration::milliseconds(300),
    failure_threshold: 3,  // 3 consecutive missed heartbeats = failure
    suspicion_level: SuspicionLevel::PhiAccrual(phi_threshold: 8.0),
    auto_demote: true,  // Failed nodes auto-demoted
});

raft.install_failure_detector(detector)?;

Automatic Failover Flow

// Listen for failover events
raft.on_failover(|event| {
    log::warn!("Failover: {:?}", event);

    match event {
        FailoverEvent::LeaderDown { old_leader, new_leader, duration } => {
            log::warn!("Leader switch: {} -> {}, duration {:?}",
                old_leader, new_leader, duration);

            // Notify clients to reconnect
            notify_clients(new_leader);
        }
        FailoverEvent::NodeRecovered { node } => {
            log::info!("Node recovered: {}", node);
            // Automatically rejoin cluster
        }
        FailoverEvent::QuorumLost { available, required } => {
            log::error!("Majority lost: {}/{}, cluster unavailable", available, required);
            // Trigger alert
        }
    }
});

Client Retry

use eneros_raft_failover::RaftClient;

// Client automatically handles Leader redirection
let client = RaftClient::new(&cluster_endpoints)
    .retry_policy(RetryPolicy::Exponential {
        initial: Duration::milliseconds(50),
        max: Duration::seconds(2),
        multiplier: 2.0,
    })
    .leader_redirect(true);

// Write with automatic retry
let result = client.write(proposal).await?;
// Internally handles NotLeader errors and redirection

Failover Time Breakdown

5. Membership Changes

Supports online membership changes, allowing nodes to be added or removed without downtime. Membership changes use single-node changes (simplified Joint Consensus), changing only one node at a time to avoid configuration conflicts.

Online Scaling

// Add new node node-6
raft.add_member(NodeId::from("node-6")).await?;

// Wait for new node to catch up on logs
raft.wait_for_catch_up(NodeId::from("node-6"), Duration::minutes(5)).await?;

// Remove node node-5
raft.remove_member(NodeId::from("node-5")).await?;

Membership Change Safety

Improvements

• Topology Engine: Topology state changes now replicated through Raft, all nodes have consistent topology views
• Digital Twin: Twin snapshots support Raft replication, state preserved after failover
• Tenant Configuration: Tenant quotas and RBAC policies synced through Raft, configuration changes take effect globally
• Guardian System: Security policies replicated through Raft, ensuring consistent policies across all nodes

Bug Fixes

• Fixed eneros-raft log conflict issue after network partition recovery (#1608)
• Fixed eneros-raft-log write blocking during snapshot installation (#1614)
• Fixed eneros-raft-election pre-vote handling issue (#1620)
• Fixed eneros-raft-failover client retry storm during frequent Leader switches (#1626)

Breaking Changes

• Topology::update_bus: Now returns RaftResult, containing commit index information
• Tenant::set_quota: Changed to async, requires .await for Raft commit
• Guardian::add_policy: Replicated through Raft, requires majority confirmation

Performance Improvements

• Cluster availability improved from 99.5% to 99.995%
• Failover time reduced from manual 5-30 minutes to automatic 800ms
• Write latency increased by only 2.5ms (including replication overhead)
• 5-node cluster write throughput reaches 30K/s

Contributors

This version was jointly completed by 25 contributors with 432 commits. Special thanks to:

• @raft-implementer: Raft algorithm core implementation
• @election-master: Leader election and pre-vote optimization
• @log-replication: Log replication and compaction
• @failover-guru: Failure detection and automatic failover

Upgrade Guide

Upgrading from v0.15.0

1. Update Dependencies

# Cargo.toml
[dependencies]
eneros-raft = { version = "0.16" }
eneros-raft-log = { version = "0.16" }
eneros-raft-election = { version = "0.16" }
eneros-raft-failover = { version = "0.16" }

2. Deploy Raft Cluster

# eneros.toml
[raft]
enabled = true
cluster_id = "eneros-prod"
peers = ["node-1", "node-2", "node-3", "node-4", "node-5"]

[raft.election]
timeout_ms = [150, 300]
heartbeat_ms = 50

[raft.log]
storage = "/var/eneros/raft"
snapshot_interval_minutes = 10
snapshot_threshold = 10000

3. Single Node to Cluster Migration

use eneros_raft::migrate;

// Migrate single-node data to Raft cluster
migrate::from_single_node("/var/eneros/data", &raft_cluster).await?;

// Verify data consistency
migrate::verify(&raft_cluster).await?;

4. Cluster Size Recommendations

Choose appropriate cluster size based on availability requirements:

Production environments recommend starting with 5 nodes, balancing availability and performance. For cross-datacenter deployment, 3 datacenters with 5 nodes (2-2-1 distribution) is recommended, ensuring no data loss during single datacenter failure.