Real-Time Determinism

Real-Time Determinism is the performance guarantee of EnerOS: through the Dual-Execution Architecture, it simultaneously satisfies the flexibility of AI inference and the determinism of power protection logic. The general execution domain handles non-real-time tasks (Agent orchestration, AI inference, business logic), while the real-time execution domain handles hard real-time tasks (relay protection, switch operations, emergency control). The real-time domain guarantees microsecond-level response with zero jitter.

Design Motivation

Real-Time Requirements of Power Systems

Different tasks in power systems have vastly different latency requirements:

Traditional single execution domain architectures cannot simultaneously satisfy microsecond-level protection and second-level AI inference: protection tasks would be blocked by AI inference, and AI inference would be interrupted by protection tasks.

Dual-Execution Architecture Solution

EnerOS divides the execution domain into two isolated layers:

┌──────────────────────────────────────────────────────────────┐
│                    General Execution Domain                   │
│                                                              │
│  · Agent orchestration · AI inference · Business logic · API │
│  · Scheduling policy: CFS (Completely Fair Scheduler)        │
│  · Latency characteristics: Probabilistic (1-50ms)           │
│  · CPU isolation: Core 0-2                                   │
└──────────────────────────┬───────────────────────────────────┘
                           │
                           ▼
                  ┌─────────────────┐
                  │  Safety Gateway │
                  │  · Command interception     │
                  │  · Constraint validation     │
                  │  · Feasibility projection    │
                  │  · Priority sorting          │
                  └────────┬────────┘
                           │
                           ▼
┌──────────────────────────────────────────────────────────────┐
│                    Real-Time Execution Domain                 │
│                                                              │
│  · Protection logic · Switch operations · Emergency control  │
│  · Watchdog                                                  │
│  · Scheduling policy: FIFO + static priority                 │
│  · Latency characteristics: Deterministic (P99 < 1ms)        │
│  · CPU isolation: Core 3 (isolated core)                     │
│  · Kernel: PREEMPT_RT or bare metal                          │
└──────────────────────────────────────────────────────────────┘

Dual-Execution Architecture Details

General Execution Domain

The general execution domain runs the standard Linux scheduler, handling latency-insensitive tasks:

use eneros_os::domain::{GeneralDomain, DomainConfig};

let general = GeneralDomain::new(DomainConfig {
    cpu_affinity: vec![0, 1, 2],          // Bind Core 0-2
    scheduler: Scheduler::CFS,
    memory_limit_mb: 4096,                 // 4GB memory limit
    io_priority: IoPriority::BestEffort,
});

// Run Agent in general domain
general.spawn(async move {
    agent.run().await
}).await?;

Real-Time Execution Domain

The real-time execution domain runs a real-time scheduler, handling hard real-time tasks:

use eneros_os::domain::{RealTimeDomain, RealTimeConfig};

let rt = RealTimeDomain::new(RealTimeConfig {
    cpu_affinity: vec![3],                // Exclusive Core 3
    scheduler: Scheduler::Fifo,
    priority_range: (50, 99),              // Real-time priority range
    memory_prealloc_mb: 256,               // Pre-allocate 256MB
    mlock: true,                           // Lock memory, disable paging
    isolated: true,                        // CPU isolation
});

// Run protection logic in real-time domain
rt.spawn_realtime(move || {
    protection_loop();
}).await?;

Inter-Domain Communication

The two execution domains communicate through lock-free queues to avoid lock contention affecting real-time performance:

use eneros_os::domain::{Channel, ChannelConfig};

// Create inter-domain channel (lock-free SPSC queue)
let (tx, rx) = Channel::new(ChannelConfig {
    capacity: 1024,
    lock_free: true,                       // Lock-free
    cache_line_size: 64,                   // Cache line alignment
});

// General domain sends command
general.spawn(async move {
    tx.send(Command::OpenBreaker(breaker_id)).await?;
}).await?;

// Real-time domain receives and executes
rt.spawn_realtime(move || {
    loop {
        match rx.recv_nonblocking() {
            Some(cmd) => execute_command(cmd),
            None => cpu_relax(),
        }
    }
}).await?;

Real-Time Domain Scheduling Strategy

Priority Levels

The real-time domain uses SCHED_FIFO scheduling with the following priority levels:

Task Registration

use eneros_os::realtime::{RealTimeTask, Priority, Schedule};

// Register protection logic task (highest priority)
rt.register_task(RealTimeTask::new(
    "protection_main",
    Priority::new(96),
    Schedule::Periodic { period_us: 1000 }, // 1ms period
    move || {
        protection_loop();
    },
)).await?;

// Register emergency control task
rt.register_task(RealTimeTask::new(
    "emergency_control",
    Priority::new(92),
    Schedule::Sporadic { deadline_us: 5000 }, // 5ms deadline
    move || {
        emergency_handler();
    },
)).await?;

// Start real-time domain
rt.start().await?;

Priority Inversion Handling

Priority Inversion Problem

When a low-priority task holds a resource needed by a high-priority task, the high-priority task gets blocked:

Time →
High priority task H:     ████████waiting for resource████████execute
Medium priority task M:               ████████████████
Low priority task L:     ██holds resource██

Solution: Priority Inheritance Protocol

EnerOS implements the Priority Inheritance Protocol in the real-time domain:

use eneros_os::realtime::{PriorityInheritance, RtMutex};

// Mutex with priority inheritance enabled
let mutex = RtMutex::new_with_inheritance(resource);

// Low priority task acquires lock, temporarily promoted to waiter's priority
rt.spawn_realtime(move || {
    let _guard = mutex.lock();  // Automatically promotes priority
    use_resource(&mut resource);
    // Restores original priority when lock is released
}).await?;

// High priority task waits for lock
rt.spawn_realtime(move || {
    let _guard = mutex.lock();  // Blocks, but low priority task has been promoted
    use_resource(&mut resource);
}).await?;

Configuration Options

use eneros_os::realtime::RtConfig;

let config = RtConfig::new()
    .priority_inheritance(true)            // Enable priority inheritance
    .priority_ceiling(false)               // Priority ceiling (more aggressive)
    .max_blocked_us(100)                   // Maximum blocking time 100μs
    .watchdog_timeout_ms(10)               // Watchdog timeout 10ms
    .cpu_isolation(true)                   // CPU isolation
    .disable_irq_balance(true);            // Disable IRQ load balancing

Interrupt Response

Interrupt Handling Layers

┌─────────────────────────────────────┐
│  Hardware Interrupt (IRQ)           │  ← Hardware triggered, < 1μs
│  · Fault detection                  │
│  · Timer                            │
│  · Network card receive             │
├─────────────────────────────────────┤
│  Top Half                           │  ← Atomic execution, < 10μs
│  · Emergency operations (e.g., trip)│
│  · Wake up real-time task           │
├─────────────────────────────────────┤
│  Bottom Half                        │  ← Real-time task, < 100μs
│  · Detailed analysis                │
│  · Log recording                    │
│  · Notify general domain            │
└─────────────────────────────────────┘

Interrupt Registration

use eneros_os::realtime::{IrqHandler, IrqConfig};

// Register fault detection interrupt
rt.register_irq(IrqHandler::new(
    IrqConfig::new(irq_number)
        .priority(99)                      // Highest priority
        .threaded(false),                  // Non-threaded (top half execution)
    move |irq| {
        // Top half: emergency trip
        trip_breaker(fault_breaker_id);
        // Wake up real-time task to handle details
        rt.wake_task("fault_handler");
        IrqResult::Handled
    },
)).await?;

// Register fault handling real-time task
rt.register_task(RealTimeTask::new(
    "fault_handler",
    Priority::new(97),
    Schedule::Sporadic { deadline_us: 1000 },
    move || {
        let fault = read_fault_details();
        log_fault(&fault);
        notify_general_domain(fault);
    },
)).await?;

Watchdog Mechanism

Soft Watchdog

EnerOS has a built-in soft watchdog in the real-time domain to monitor task execution:

use eneros_os::realtime::{Watchdog, WatchdogConfig};

let watchdog = Watchdog::new(WatchdogConfig {
    timeout_ms: 10,                        // 10ms timeout
    action_on_timeout: WatchdogAction::Reboot, // Reboot on timeout
    stats_interval_ms: 1000,
});

// Task periodically feeds the watchdog
rt.register_task(RealTimeTask::new(
    "protection_main",
    Priority::new(96),
    Schedule::Periodic { period_us: 1000 },
    move || {
        watchdog.kick();                   // Feed the watchdog
        protection_loop();
    },
)).await?;

watchdog.start().await?;

Hardware Watchdog

Supports hardware watchdog to ensure recovery even when the kernel is completely frozen:

use eneros_os::realtime::HardwareWatchdog;

let hw_watchdog = HardwareWatchdog::open("/dev/watchdog")?
    .timeout_ms(100)                       // 100ms timeout
    .auto_kick(true);                      // Real-time domain auto-feeds

hw_watchdog.start()?;

Watchdog Timeout Handling

Safety Gateway

The safety gateway is the bridge between the two execution domains, responsible for safe command delivery:

use eneros_os::gateway::{SafetyGateway, GatewayConfig, Priority};

let gateway = SafetyGateway::new(GatewayConfig {
    constraint_engine: constraints.clone(),
    channel_capacity: 1024,
    max_command_rate: 10000,                // Max commands per second
    enable_projection: true,                // Enable feasibility projection
});

// Agent dispatches commands through the gateway
let result = gateway.execute(
    Command::SetGeneration { bus: 1, mw: 80.0 },
    Priority::Normal,
).await?;

match result {
    ExecuteResult::Executed => {
        log::info!("Command executed");
    }
    ExecuteResult::Projected(projected) => {
        log::warn!("Command was projected: {:?}", projected);
    }
    ExecuteResult::Rejected(reason) => {
        log::error!("Command rejected: {}", reason);
    }
    ExecuteResult::Queued => {
        log::info!("Command queued, waiting for real-time domain processing");
    }
}

Emergency Command Priority

// Emergency commands (e.g., protection trip) are prioritized
gateway.execute(
    Command::TripBreaker(fault_breaker_id),
    Priority::Emergency,                    // Emergency priority
).await?;
// Emergency commands reach the real-time domain within < 100μs

Real-Time Performance Metrics

Latency Metrics

Throughput Metrics

Scheduling Latency Test

use eneros_os::realtime::{LatencyTest, LatencyResult};

// Run scheduling latency test
let result: LatencyResult = rt.test_latency(Duration::seconds(60)).await?;

println!("Average latency: {} μs", result.avg_us);
println!("P99 latency: {} μs", result.p99_us);
println!("P99.9 latency: {} μs", result.p999_us);
println!("Max latency: {} μs", result.max_us);
println!("Violation count: {}", result.violations);

Complete Example: Protection Logic Real-Time Execution

use eneros_os::{domain::{RealTimeDomain, RealTimeConfig}, realtime::*};
use eneros_constraint::ConstraintEngine;
use std::time::Duration;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1. Create real-time domain
    let rt = RealTimeDomain::new(RealTimeConfig {
        cpu_affinity: vec![3],
        scheduler: Scheduler::Fifo,
        priority_range: (50, 99),
        memory_prealloc_mb: 256,
        mlock: true,
        isolated: true,
    });

    // 2. Enable priority inheritance
    rt.set_config(RtConfig::new()
        .priority_inheritance(true)
        .max_blocked_us(100)
        .watchdog_timeout_ms(10)
        .cpu_isolation(true));

    // 3. Register fault detection interrupt
    rt.register_irq(IrqHandler::new(
        IrqConfig::new(IRQ_FAULT_DETECT)
            .priority(99)
            .threaded(false),
        move |_irq| {
            trip_breaker(FAULT_BREAKER_ID);
            rt.wake_task("fault_handler");
            IrqResult::Handled
        },
    ))?;

    // 4. Register protection logic periodic task
    rt.register_task(RealTimeTask::new(
        "protection_main",
        Priority::new(96),
        Schedule::Periodic { period_us: 1000 },
        move || {
            watchdog.kick();
            protection_loop();
        },
    ))?;

    // 5. Register fault handling task
    rt.register_task(RealTimeTask::new(
        "fault_handler",
        Priority::new(97),
        Schedule::Sporadic { deadline_us: 1000 },
        move || {
            let fault = read_fault_details();
            log_fault(&fault);
            notify_general_domain(fault);
        },
    ))?;

    // 6. Start watchdog
    let watchdog = Watchdog::new(WatchdogConfig {
        timeout_ms: 10,
        action_on_timeout: WatchdogAction::Reboot,
        stats_interval_ms: 1000,
    });
    watchdog.start()?;

    // 7. Start real-time domain
    rt.start()?;

    Ok(())
}

Comparison with RT-Linux

Comparison with Other Real-Time Solutions

Limitations and Trade-offs

Next Steps

• Power-Native First - Power-Native First design philosophy
• Constraint as Law - Real-time domain constraint validation
• Safety Guard - Safety gateway details
• Real-Time Dual Execution Domain - Complete real-time domain capabilities