Dual-Execution Architecture

EnerOS adopts a Dual-Execution Architecture that separates general computing from real-time control, simultaneously satisfying the flexibility of AI inference and the determinism of protection logic. This architecture draws on the dual-kernel idea of RT-Linux but is implemented entirely in Rust user space, achieving domain isolation and communication through CPU affinity, SCHED_FIFO scheduling, shared memory channels, and lock-free queues.

Design Motivation

Power systems simultaneously have two types of workloads with completely different requirements for latency, jitter, and determinism:

Putting both in the same scheduling domain would lead to: GC/allocation/network IO jitter from AI inference directly polluting protection logic, violating the NERC PRC-024 100ms immunity requirement. EnerOS eliminates this conflict at its root through the dual-execution architecture.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                         Application Layer                            │
│   DispatchAgent · OperationAgent · TradingAgent · SelfHealingAgent  │
├─────────────────────────────────────────────────────────────────────┤
│                       Agent Runtime                                  │
│      eneros-agent · eneros-reasoning · eneros-memory · eneros-tool   │
├──────────────┬──────────────────────┬───────────────────────────────┤
│  General     │     Safety Gateway    │       Real-Time               │
│  Execution   │                       │       Execution Domain        │
│  Domain      │                       │                               │
│              │                       │                               │
│ · Agent      │ · Command             │ · Protection logic            │
│   orchestration│   interception      │   (overcurrent/overvoltage/   │
│ · LLM        │ · Constraint          │    frequency)                 │
│   inference  │   validation          │ · Switch operations           │
│ · Business   │ · Feasibility         │   (trip/close)                │
│   logic      │   projection          │ · Emergency control           │
│ · REST API   │ · Priority sorting    │   (generator/load shedding/   │
│ · Web service│ · Audit logging       │    islanding)                 │
│              │                       │ · Real-time data acquisition  │
│              │                       │   (PMU/RTU)                   │
│              │                       │ · P99 < 1ms · Jitter < 100μs  │
│              │                       │                               │
│ SCHED_OTHER  │  Bidirectional bridge │ SCHED_FIFO + CPU isolated     │
│ CFS scheduling│  Lock-free queue +   │ mlockall + huge pages         │
│              │  shared memory        │                               │
├──────────────┴──────────────────────┴───────────────────────────────┤
│                    Power-Native Kernel                               │
│       topology · powerflow · constraint · equipment · timeseries     │
├─────────────────────────────────────────────────────────────────────┤
│                       Infrastructure                                 │
│           eneros-os · eneros-multiregion · eneros-tenant             │
└─────────────────────────────────────────────────────────────────────┘

General Execution Domain

Responsibilities

The general domain carries all non-real-time, jitter-tolerant workloads:

• Agent lifecycle management: Creation, suspension, resumption, destruction, migration
• AI/LLM inference: Calling local or remote large models for planning, explanation, natural language interaction
• Business logic processing: Economic dispatch algorithms, bidding strategies, load forecasting post-processing
• REST/GraphQL API services: Exposing HTTP/gRPC interfaces externally
• Dashboard web services: Visualization frontend and SSE push
• Report generation: PDF/Excel/Word export and scheduled emails

Scheduling Strategy

The general domain uses the standard Linux CFS (Completely Fair Scheduler) scheduler with the SCHED_OTHER scheduling policy (policy=0), supporting multi-threaded concurrency and async IO (tokio runtime). No CPU isolation requirements, can run on any core.

Latency Characteristics

Key Crates

Real-Time Execution Domain

Responsibilities

The real-time domain carries all hard real-time, determinism-requiring workloads:

• Protection logic: Overcurrent, overvoltage, undervoltage, frequency anomaly, distance protection
• Switch operations: Tripping, closing, reclosing
• Emergency control: Generator shedding, load shedding, islanding, AGC
• Real-time data acquisition: PMU synchronized phasors, RTU telemetry, SOE events
• Real-time control closed-loop: Voltage regulation, reactive compensation, active power distribution

Scheduling Strategy

The real-time domain adopts the SCHED_FIFO (policy=1) real-time scheduling policy, combined with the following mechanisms to guarantee determinism:

Latency Characteristics

Key Crates

Inter-Domain Isolation Mechanisms

CPU Isolation

The general domain and real-time domain run on different CPU cores, enforced through sched_setaffinity:

use eneros_os::rt::{RtConfig, RtRuntime};

// Real-time domain config: bind to CPU 2, 3, SCHED_FIFO priority 80
let rt_config = RtConfig {
    cpus: vec![2, 3],
    priority: 80,
    lock_memory: true,
    use_huge_pages: true,
};

let rt_runtime = RtRuntime::new(rt_config);
rt_runtime.configure_current_thread()?;

Memory Isolation

• mlockall: Real-time domain locks all memory pages, disabling paging
• Huge pages: Uses 2MB huge pages to reduce TLB misses
• Memory pool: Real-time domain uses eneros-perf’s arena allocator, zero runtime allocation

Fault Isolation

• Real-time domain process crash does not affect general domain (independent processes)
• General domain GC/allocation does not block real-time domain (independent cores)
• Safety gateway’s watchdog monitors real-time domain heartbeat, triggers fail-safe on timeout

Communication Mechanisms

1. Lock-Free SPSC Queue (Same Process)

eneros-os::rt::ipc::RtCommandQueue provides a lock-free single-producer single-consumer queue within the same process:

use eneros_os::rt::ipc::RtCommandQueue;
use eneros_core::Command;

// Lock-free command queue with capacity 1024
let queue: RtCommandQueue<Command, 1024> = RtCommandQueue::new();

// General domain thread (producer): try_push
let cmd = Command::new_switch_toggle(42, true, "operator-1");
match queue.try_push(cmd) {
    Ok(()) => println!("Command enqueued"),
    Err(cmd) => println!("Queue full, command rejected: {:?}", cmd),
}

// Real-time domain thread (consumer): try_pop
match queue.try_pop() {
    Some(cmd) => {
        println!("Real-time domain received command: {:?}", cmd);
        // Deterministic execution of protection logic
    }
    None => {
        // Queue empty, return immediately (non-blocking)
    }
}

Features:

• Single try_push / try_pop latency < 100ns
• Lock-free, no CAS, no system calls (only Release/Acquire atomic operations)
• Available capacity = CAPACITY - 1 (reserve 1 slot to distinguish empty/full)
• Thread safety requirement: single producer + single consumer

2. Shared Memory Channel (Cross-Process)

eneros-os::rt::shm::SharedMemoryChannel<T> provides a typed shared memory channel across processes:

use eneros_os::rt::shm::SharedMemoryChannel;
use eneros_core::Command;

// Create shared memory channel (capacity 256, element type Command)
let channel = SharedMemoryChannel::<Command>::create(
    "/eneros-rt-cmd",  // Shared memory name
    256,                // Capacity
)?;

// Producer process (general domain): push
channel.push(&Command::new_switch_toggle(42, true, "operator-1"))?;

// Consumer process (real-time domain): blocking_pop (with eventfd notification)
let cmd = channel.blocking_pop(Duration::from_millis(10))?;
println!("Real-time domain received command: {:?}", cmd);

Features:

• Based on mmap + eventfd + atomic index
• Single push + pop latency < 5μs
• Cross-process zero-copy (T: Copy bitwise copy)
• Magic number validation (SHM_MAGIC = 0x454E4552) prevents misuse

3. Priority Command Queue (Safety Gateway)

eneros-gateway::priority_queue::SharedPriorityCommandQueue provides a cross-domain command queue with priority:

use eneros_gateway::priority_queue::SharedPriorityCommandQueue;
use eneros_core::{Command, CommandPriority};

let queue = std::sync::Arc::new(SharedPriorityCommandQueue::new(1024));

// Emergency command (protection trip): Critical priority
let trip_cmd = Command::new_switch_toggle(42, false, "protection")
    .with_priority(CommandPriority::Critical);
queue.enqueue(trip_cmd);

// Normal command (dispatch plan): Normal priority
let dispatch_cmd = Command::new_generator_setpoint(1, 50.0, "dispatcher")
    .with_priority(CommandPriority::Normal);
queue.enqueue(dispatch_cmd);

// Real-time executor dequeues by priority (Critical before Normal)

Safety Gateway

The safety gateway is the only bridge between the two execution domains, ensuring all commands from the general domain to the real-time domain pass through safety validation.

Workflow

General domain Agent generates command
        │
        ▼
┌───────────────────┐
│  SafetyGateway    │
│  1. Command        │
│     interception   │
│  2. Constraint     │ ← eneros-constraint
│     validation     │
│  3. Priority       │
│     sorting        │
│  4. Feasibility    │ ← Project to nearest feasible solution
│     projection     │   when constraints violated
│  5. Audit logging  │ ← eneros-audit (WORM)
└─────┬─────────────┘
      │
      ├── Pass ──► Priority queue ──► Real-time domain execution
      │
      └── Violation ──► Project / Reject ──► Notify Agent

Decision Matrix

Complete Code Example

use eneros_gateway::{SafetyGateway, command::*, executor::DeviceCommandExecutor};
use eneros_device::DeviceManager;
use std::sync::Arc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Prepare device manager (real device execution)
    let device_manager = Arc::new(DeviceManager::new());

    // 2. Create command executor
    let executor = Arc::new(DeviceCommandExecutor::new(device_manager.clone()));

    // 3. Create safety gateway (history capacity 1000)
    let gateway = SafetyGateway::with_executor(1000, executor);

    // 4. Register safety checks (constraint engine, permissions, limits)
    gateway.register_safety_check(Box::new(VoltageLimitCheck::new(0.95, 1.05)));
    gateway.register_safety_check(Box::new(CurrentLimitCheck::new(1000.0)));
    gateway.register_safety_check(Box::new(PermissionCheck::new("dispatcher")));

    // 5. Construct control command: close switch 42
    let cmd = Command::new(
        CommandType::SwitchToggle,
        42,
        CommandPriority::Normal,
        "operator-1",
    )
    .with_parameter("closed", 1.0);

    // 6. Execute through safety gateway
    match gateway.execute_command(cmd).await {
        Ok(()) => {
            println!("Command executed successfully");
            let history = gateway.command_history();
            println!("History record count: {}", history.len());
        }
        Err(e) => {
            println!("Command execution failed: {}", e);
        }
    }

    // 7. Batch execution (same device serial, different device parallel)
    let batch = vec![
        Command::new(CommandType::SwitchToggle, 1, CommandPriority::Normal, "op")
            .with_device_id("rtu-1"),
        Command::new(CommandType::SwitchToggle, 2, CommandPriority::Normal, "op")
            .with_device_id("rtu-2"),
        Command::new(CommandType::SwitchToggle, 3, CommandPriority::Normal, "op")
            .with_device_id("rtu-1"),
    ];
    let results = gateway.execute_command_batch(batch).await;
    for (i, result) in results.iter().enumerate() {
        println!("Command {} execution result: success={}", i, result.success);
    }

    Ok(())
}

/// Voltage limit violation check
struct VoltageLimitCheck {
    v_min: f64,
    v_max: f64,
}

impl VoltageLimitCheck {
    fn new(v_min: f64, v_max: f64) -> Self {
        Self { v_min, v_max }
    }
}

impl SafetyCheck for VoltageLimitCheck {
    fn validate(&self, command: &Command) -> eneros_core::Result<()> {
        if let Some(v) = command.parameters.get("voltage") {
            if *v < self.v_min || *v > self.v_max {
                return Err(eneros_core::EnerOSError::Constraint(format!(
                    "Voltage {} exceeds allowable range [{}, {}]",
                    v, self.v_min, self.v_max
                )));
            }
        }
        Ok(())
    }

    fn name(&self) -> &str {
        "VoltageLimitCheck"
    }

    fn description(&self) -> &str {
        "Validate voltage is within allowable range"
    }
}

struct CurrentLimitCheck {
    i_max: f64,
}

impl CurrentLimitCheck {
    fn new(i_max: f64) -> Self {
        Self { i_max }
    }
}

impl SafetyCheck for CurrentLimitCheck {
    fn validate(&self, command: &Command) -> eneros_core::Result<()> {
        if let Some(i) = command.parameters.get("current") {
            if *i > self.i_max {
                return Err(eneros_core::EnerOSError::Constraint(format!(
                    "Current {} exceeds maximum allowable value {}",
                    i, self.i_max
                )));
            }
        }
        Ok(())
    }

    fn name(&self) -> &str {
        "CurrentLimitCheck"
    }

    fn description(&self) -> &str {
        "Validate current does not exceed maximum allowable value"
    }
}

struct PermissionCheck {
    required_role: String,
}

impl PermissionCheck {
    fn new(role: &str) -> Self {
        Self {
            required_role: role.to_string(),
        }
    }
}

impl SafetyCheck for PermissionCheck {
    fn validate(&self, command: &Command) -> eneros_core::Result<()> {
        if command.operator != self.required_role && command.operator != "admin" {
            return Err(eneros_core::EnerOSError::Gateway(format!(
                "Operator {} has no permission to execute this command (required role: {})",
                command.operator, self.required_role
            )));
        }
        Ok(())
    }

    fn name(&self) -> &str {
        "PermissionCheck"
    }

    fn description(&self) -> &str {
        "Validate operator permissions"
    }
}

Lock-Free Optimization Path

To eliminate parking_lot::RwLock lock contention, EnerOS v0.42.0 introduced a lock-free safety gateway path based on atomic state machine + SkipList, pushing the hot path P99 from 4-8ms to < 1ms:

Command
    │
    ▼
LockFreeGatewayState::try_acquire_device(device_id, Idle → Validating)
    │  CAS AtomicU8 (per-device atomic state machine)
    ▼
ParallelSafetyChecker::check_parallel(cmd)
    │  rayon::scope executes all SafetyChecks in parallel
    ▼
AtomicCommandHistory::insert(cmd)
    │  crossbeam_skiplist::SkipMap lock-free insertion (sorted by LSN)
    ▼
LockFreeGatewayState::release_device(device_id, Idle)

Device State Machine

use eneros_gateway::lockfree::{LockFreeGatewayState, DeviceState};

let state = LockFreeGatewayState::new();

// CAS: Idle → Validating (success means device acquired)
state.try_acquire_device("rtu-1", DeviceState::Idle, DeviceState::Validating)?;

// Execute safety validation...

// Release: Validating → Idle
state.release_device("rtu-1", DeviceState::Idle);

// Query device state
match state.device_state("rtu-1") {
    DeviceState::Idle => println!("Device idle"),
    DeviceState::Validating => println!("Device validating"),
    DeviceState::Executing => println!("Device executing"),
    DeviceState::Cooldown => println!("Device cooling down"),
}

Scheduling Strategy Comparison

Real-Time Domain Guarantee Mechanisms

1. CPU Isolation

Through kernel boot parameters isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3, CPU 2 and 3 are completely isolated, and kernel threads will not be scheduled to these cores.

2. SCHED_FIFO Scheduling

Real-time domain threads use the SCHED_FIFO scheduling policy with priority 80 (higher than all non-real-time threads), and will not be preempted by the general domain.

3. Memory Locking

mlockall(MCL_CURRENT | MCL_FUTURE) locks all current and future memory pages, disabling paging and eliminating page faults (page faults can take 10ms+).

4. Huge Pages

Uses 2MB huge pages to reduce TLB misses, combined with madvise(MADV_HUGEPAGE) to enable transparent huge pages.

5. Watchdog

use eneros_os::rt::watchdog::Watchdog;

// Real-time domain watchdog: 1ms timeout
let mut watchdog = Watchdog::new(Duration::from_millis(1));
watchdog.start();

loop {
    // Real-time task loop
    execute_protection_logic();
    watchdog.kick(); // Feed the watchdog
}

Performance Metrics

Measured data in a standard test environment (4 cores / 8GB / Ubuntu 22.04, CPU 2-3 isolated):

Applicable Scenarios

Must Use Real-Time Domain

• Relay protection logic (overcurrent, distance, differential)
• Emergency control (low-frequency load shedding, generator shedding, islanding)
• AGC closed-loop control
• PMU synchronized phasor acquisition
• Switch trip / close operations

Can Use General Domain

• Dispatch plan generation
• Load forecasting
• Electricity trading bidding
• Distribution network planning
• Report generation
• Web visualization

Hybrid Scenarios (Dual-Domain Collaboration)

• Fault self-healing: General domain locates fault + real-time domain executes isolation and recovery
• Intelligent dispatch: General domain generates plan + real-time domain dispatches instructions
• Digital twin: General domain simulation + real-time domain mirrors field state

Related Documentation

• Layered Architecture - Five-layer architecture details
• Real-Time Dual Execution Domain - Capability details
• Safety Guard - Safety gateway capabilities
• Crate Dependencies - Crate dependency graph