ADR-0007 Zero Trust & mTLS Enforcement

• Status: Accepted
• Date: 2024-09
• Decision Makers: EnerOS Security Architecture Group
• Related Capabilities: Zero Trust & Security Enhancement, Compliance Mapping
• Related ADR: ADR-0002 Power-Native AgentOS

Context

Decision Timing

In the second half of 2024, EnerOS was about to enter the provincial grid pilot deployment phase. The pilot environment involved three types of production areas: dispatch centers, substations, and power plants, requiring simultaneous compliance with multiple stringent security standards. In a security assessment before pilot launch, a third-party testing organization pointed out that EnerOS v0.30’s internal communication used plaintext gRPC, relying on network boundaries (firewalls + VLANs) for isolation, posing lateral movement risks and failing the MLPS 2.0 Level 3 assessment. This ADR was initiated to address this assessment finding.

Compliance Requirements

EnerOS is deployed in power critical infrastructure and must simultaneously satisfy the following three compliance standards:

The three standards have common requirements in the following areas:

1. Communication confidentiality and integrity: All control command transmissions must be encrypted and tamper-proof
2. Mutual identity authentication: Both communicating parties must mutually verify identity, not one-way
3. Fine-grained access control: Authorization based on identity and context, not network location
4. Complete audit chain: Each remote command can be traced back to the issuing identity and time
5. Continuous verification: No implicit trust; every access requires verification

Current Pain Points

Before EnerOS v0.30, internal services used plaintext gRPC, with the following issues:

Pain Point 1: Lateral Movement Once Boundary is Breached

┌─────────────────────────────────────────┐
│  Dispatch Center VLAN (trusted)         │
│  ┌────────┐  plaintext gRPC  ┌────────┐ │
│  │ Agent  │ ──────────► │Topology│ │
│  │ Runtime│             │ Engine │ │
│  └────────┘             └────────┘ │
│       │  Once attacker enters VLAN    │
│       ▼  can sniff/forge RPC          │
└─────────────────────────────────────────┘

Pain Point 2: Broken Audit Chain

Plaintext RPC cannot prove command origin; audit logs can only record “a certain IP performed a certain operation” and cannot bind to specific identities. In case of incidents, forensics cannot prove “which engineer/Agent issued the command.”

Pain Point 3: Non-compliant

MLPS 2.0 Level 3 requires “communication integrity” and “identity authentication”; plaintext gRPC directly fails to comply. The testing organization explicitly required remediation before the pilot.

Specific Event Triggering the Decision

In August 2024, a third-party testing organization conducted an MLPS pre-assessment of EnerOS v0.30, issuing a “communication security non-compliant” finding requiring remediation within 2 months, otherwise the pilot could not proceed. This ADR was urgently initiated in response to this finding.

Constraints at Decision Time

Decision

Implement Zero Trust architecture: all internal inter-service communication enforces mTLS mutual authentication, with certificates issued and rotated by a built-in CA; audit uses HMAC chained structure written to WORM (Write Once Read Many) storage; identity authentication binds SPIFFE ID and tenant_id, with RBAC + ABAC for authorization.

Core Rationale

Reason 1: Zero Trust fits power critical infrastructure

The core principle of Zero Trust, “Never Trust, Always Verify,” aligns with the security philosophy of power systems—every operation requires identity authentication and permission validation, without relaxation due to being in a “secure zone.”

Reason 2: mTLS is the industry standard for mutual authentication

mTLS completes mutual identity authentication during the communication handshake phase, requiring no additional application-layer coding, and is naturally compatible with gRPC and HTTP/2. Compared to application-layer encryption, mTLS is handled at the transport layer, with minimal intrusion into business code.

Reason 3: Built-in CA enables certificate lifecycle automation

Power deployment environments are typically network-isolated and cannot rely on external CA services. Built-in CA combined with auto-rotation makes certificate management transparent to operations.

Reason 4: SPIFFE provides a unified identity framework

SPIFFE (Secure Production Identity Framework for Everyone) is an identity framework designed for cloud-native and hybrid environments; its SPIFFE ID can uniquely identify workloads across services and tenants, aligning with EnerOS’s multi-tenant model.

Reason 5: HMAC chained audit is tamper-proof

Chained audit logs are immutable once written, satisfying compliance forensics requirements and proving log integrity.

Zero Trust Architecture Overview

                    ┌──────────────────────────┐
                    │   Control Plane           │
                    │  ┌────┐  ┌─────┐  ┌────┐  │
                    │  │ CA │  │SPIRE│  │PDP │  │
                    │  └────┘  └─────┘  └────┘  │
                    └──────────────────────────┘
                              │ Issues certificates
                              ▼
┌─────────── Data Plane ────────────────────┐
│                                            │
│  ┌────────┐  mTLS mutual auth  ┌────────┐  │
│  │ Agent  │ ◄─────────────► │Topology│  │
│  │ Runtime│   SPIFFE ID       │ Engine │  │
│  └────────┘   mutual verify   └────────┘  │
│      │                          │         │
│      ▼                          ▼         │
│  ┌────────┐                 ┌────────┐    │
│  │ Audit  │ ◄── HMAC chain ──► │  WORM  │ │
│  │ Logger │                 │ Store  │    │
│  └────────┘                 └────────┘    │
└────────────────────────────────────────────┘

mTLS Configuration Example

EnerOS internal service mTLS configuration is centrally managed via eneros.toml:

[security.mtls]
enabled = true                    # Enforced, cannot be disabled
min_version = "TLS1.3"            # Minimum TLS 1.3, disable old versions
cert_rotation = "24h"             # Certificate validity 24 hours, auto-rotation
cert_overlap = "1h"               # New/old certificate overlap 1 hour, smooth transition

[security.mtls.cipher_suites]
allowed = ["TLS_AES_256_GCM_SHA384", "TLS_CHACHA20_POLY1305_SHA256"]
denied = ["TLS_RSA_*", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"]  # Disable weak suites

[security.ca]
type = "internal"                 # Built-in CA
key_algorithm = "ECDSA-P256"      # Elliptic curve signature
key_storage = "hsm"               # Private key stored in HSM or soft HSM
cluster_mode = "raft"             # CA cluster uses Raft for high availability
backup_interval = "6h"            # CA data backed up every 6 hours

[security.identity]
framework = "spiffe"              # Use SPIFFE identity framework
trust_domain = "eneros.example.com"
id_format = "spiffe://{tenant}/{service}/{instance}"

Rust code-level mTLS client configuration:

use rustls::{ClientConfig, ServerConfig};
use rustls::pki_types::{CertificateDer, PrivateKeyDer};

/// Build mTLS client configuration
pub fn build_mtls_client_config(
    cert: CertificateDer<'static>,
    key: PrivateKeyDer<'static>,
    ca_cert: CertificateDer<'static>,
) -> Result<ClientConfig> {
    let mut root_store = rustls::RootCertStore::empty();
    root_store.add(ca_cert)?;

    Ok(ClientConfig::builder()
        .with_root_certificates(root_store)
        .with_client_auth_cert(vec![cert], key)?
        .with_protocol_versions(&[&rustls::version::TLS13])?  // Enforce TLS 1.3
        .build())
}

/// Build mTLS server configuration
pub fn build_mtls_server_config(
    cert: CertificateDer<'static>,
    key: PrivateKeyDer<'static>,
    ca_cert: CertificateDer<'static>,
) -> Result<ServerConfig> {
    let mut client_root = rustls::RootCertStore::empty();
    client_root.add(ca_cert)?;

    Ok(ServerConfig::builder()
        .with_no_client_auth()  // Placeholder, overridden below to enforce client auth
        .with_client_cert_verifier(Arc::new(SpiffeVerifier::new(client_root)))
        .with_single_cert(vec![cert], key)?
        .build())
}

/// Client certificate verifier based on SPIFFE ID
pub struct SpiffeVerifier {
    roots: rustls::RootCertStore,
}

impl ServerCertVerifier for SpiffeVerifier {
    fn verify_server_cert(
        &self,
        end_entity: &CertificateDer,
        intermediates: &[CertificateDer],
        server_name: &ServerName,
        now: UnixTime,
    ) -> Result<ServerCertVerified, Error> {
        // 1. Verify certificate chain signature
        self.verify_chain(end_entity, intermediates, now)?;
        // 2. Extract and verify SPIFFE ID
        let spiffe_id = extract_spiffe_id(end_entity)?;
        if !self.is_authorized(&spiffe_id) {
            return Err(Error::General(format!(
                "unauthorized SPIFFE ID: {}", spiffe_id
            )));
        }
        Ok(ServerCertVerified::assertion())
    }
}

Certificate Lifecycle Management

The built-in CA automatically manages certificate issuance, rotation, and revocation:

pub struct CertificateLifecycle {
    ca: Arc<CertificateAuthority>,
    rotation_interval: Duration,
    overlap: Duration,
}

impl CertificateLifecycle {
    /// Start certificate rotation loop
    pub async fn run_rotation_loop(self: Arc<Self>) {
        let mut ticker = tokio::time::interval(self.rotation_interval - self.overlap);
        loop {
            ticker.tick().await;
            if let Err(e) = self.rotate_certificate().await {
                tracing::error!("certificate rotation failed: {}", e);
                self.alert_ops_team(e).await;
            }
        }
    }

    /// Rotate a single service's certificate
    async fn rotate_certificate(&self) -> Result<()> {
        let new_cert = self.ca.issue_certificate(&self.service_identity()).await?;
        let new_key = self.ca.generate_key_pair()?;

        // Atomic swap: new certificate takes effect, old certificate retained for overlap period then revoked
        self.credential_store.swap(CredentialPair {
            cert: new_cert,
            key: new_key,
        })?;

        // Schedule old certificate revocation
        let ca = self.ca.clone();
        let old_serial = self.credential_store.previous_serial()?;
        tokio::spawn(async move {
            tokio::time::sleep(ca.overlap).await;
            ca.revoke(old_serial).await.ok();
        });

        Ok(())
    }
}

Identity and Authorization Model

Each internal service obtains a SPIFFE ID at startup, binding tenant_id and roles:

pub struct ServiceIdentity {
    pub spiffe_id: SpiffeId,        // spiffe://eneros.example.com/tenant-a/topology/node-3
    pub tenant_id: TenantId,
    pub service: ServiceName,
    pub instance: InstanceId,
    pub roles: Vec<Role>,           // RBAC roles
    pub attributes: Vec<Attribute>, // ABAC attributes
}

/// Authorization Decision Point (PDP)
pub struct AuthorizationPdp {
    rbac: RbacEngine,
    abac: AbacEngine,
}

impl AuthorizationPdp {
    pub fn decide(&self, identity: &ServiceIdentity, action: &Action) -> Decision {
        // First RBAC coarse-grained filter
        if !self.rbac.allow(identity, action) {
            return Decision::Deny("RBAC denied".into());
        }
        // Then ABAC fine-grained validation
        match self.abac.evaluate(identity, action) {
            AbacResult::Allow => Decision::Allow,
            AbacResult::Deny(reason) => Decision::Deny(reason),
            AbacResult::NeedMoreInfo => Decision::Deny("insufficient context".into()),
        }
    }
}

Chained Audit Log

Each audit log entry contains the HMAC digest of the previous entry, forming a chain structure that cannot be tampered with once written to WORM storage:

pub struct AuditEntry {
    pub sequence: u64,
    pub timestamp: NanosecondTimestamp,
    pub actor_spiffe_id: String,
    pub actor_tenant_id: TenantId,
    pub action: String,
    pub target: String,
    pub result: AuditResult,
    pub prev_hash: [u8; 32],   // HMAC-SHA256 of previous log entry
    pub this_hash: [u8; 32],   // HMAC-SHA256 of this log entry
}

impl AuditEntry {
    pub fn compute_hash(&self, secret: &HmacKey) -> [u8; 32] {
        let mut buf = Vec::new();
        buf.extend_from_slice(&self.sequence.to_le_bytes());
        buf.extend_from_slice(&self.timestamp.to_le_bytes());
        buf.extend_from_slice(self.actor_spiffe_id.as_bytes());
        buf.extend_from_slice(self.actor_tenant_id.as_bytes());
        buf.extend_from_slice(self.action.as_bytes());
        buf.extend_from_slice(self.target.as_bytes());
        buf.extend_from_slice(self.result.as_bytes());
        buf.extend_from_slice(&self.prev_hash);
        hmac_sha256(secret, &buf)
    }
}

Consequences

Benefits

1. Attackers cannot move laterally after boundary breach

Even if an attacker breaches a VLAN, without a valid SPIFFE ID and certificate, they cannot establish mTLS connections with any internal service. Lateral movement changes from “breach boundary and proceed” to “must compromise CA and steal private keys.”

2. Any remote command can be fully traced to the issuing identity

Each command’s audit log binds SPIFFE ID and tenant_id, combined with chained HMAC and WORM storage, satisfying the “non-repudiation” requirement for compliance forensics.

3. One technical base satisfies multiple compliance standards simultaneously

4. Certificate rotation automation reduces operations burden

Certificates auto-rotate every 24 hours; operations need not manage manually. Even if a private key leaks, the attack window is at most 24 hours.

5. Multi-tenant isolation strengthened

Each Agent and service’s identity binds tenant_id; cross-tenant access requires explicit authorization, preventing inter-tenant privilege escalation at the architectural level.

Costs

1. mTLS handshake adds approximately 5ms latency

Measured TLS 1.3 handshake (including mutual authentication) takes about 5ms, optimized to acceptable range through Session Resumption and connection reuse. Long-term connection handshake overhead amortized approaches 0.

2. Internal CA high availability requires independent assurance

CA failure prevents new certificate issuance, making it a critical single point. Guaranteed through Raft cluster + HSM backup, but still requires independent monitoring.

3. Certificate rotation process requires supporting monitoring and alerting

Rotation failure may cause services to fail communication after certificate expiry. Rotation failure alerts deployed, with 1-hour overlap period as buffer.

4. Increased debugging complexity

Viewing RPC content via packet capture requires key log configuration (SSLKEYLOGFILE); production environments require troubleshooting through audit logs rather than traffic captures.

5. Interoperability with legacy systems requires gateways

Some older IEDs (Intelligent Electronic Devices) do not support TLS 1.3 and require protocol gateways to proxy mTLS.

Follow-up Work

• Introduce IDS covering OT attack signature database (landed in v0.40)
• Support post-quantum (PQC) hybrid key agreement to address long-term threats (planned for v0.55)
• Improve HSM hardware adaptation, support Chinese national cryptographic algorithms SM2/SM3/SM4 (planned for v0.50)
• Introduce SPIRE federation for multi-cluster cross-trust-domain communication (planned for v0.60)
• See Zero Trust Architecture & mTLS compliance mapping

Alternatives

Option A: Service Mesh Sidecar (e.g., Istio)

Description: Introduce Istio service mesh; Sidecar proxies automatically handle mTLS, with application code unaware.

Advantages:

• Zero application code changes; mTLS completely transparent
• Provides rich traffic management, circuit breaking, observability capabilities
• Mature community, comprehensive documentation

Rejection Reasons:

• Sidecar adds 1-3ms latency per hop; unacceptable for real-time domain (< 100μs)
• Introduces Envoy dependency, binary size increases 100MB+
• Istio control plane complexity; difficult for power operations teams to assume
• Conflicts with EnerOS’s kernel-native philosophy—security should be in the kernel, not Sidecar

Option B: Application-Layer Encryption

Description: Each RPC call self-encrypts and signs; application code explicitly handles keys and authentication.

Advantages:

• Does not depend on transport layer; high flexibility
• Can finely control encryption policy per message

Rejection Reasons:

• Each RPC reinvents the wheel; error-prone (key management, random numbers, protocol design)
• Large application code intrusion; each RPC requires 5-10 lines of encryption code
• Poor performance; application-layer encryption cannot leverage TLS hardware acceleration
• Difficult to unify audit; encryption implementation scattered

Option C: Retain Boundary + Hardening

Description: Maintain existing plaintext gRPC, improve boundary security through hardened firewall rules, VLAN isolation, intrusion detection, etc.

Advantages:

• Minimal changes; low development cost
• No new components; operations familiar

Rejection Reasons:

• Cannot address internal threats and zero-day vulnerabilities; boundary breach means total compromise
• MLPS 2.0 Level 3 explicitly requires communication encryption; this approach cannot pass assessment
• Audit chain still relies on IP; cannot bind identity
• Runs counter to Zero Trust philosophy

Option D: WireGuard Full Tunnel

Description: Use WireGuard to establish encrypted tunnels between nodes; all internal traffic goes through tunnels.

Advantages:

• Excellent performance; WireGuard kernel-mode implementation, latency increase < 1ms
• Simple configuration; lightweight key management
• Modern encryption protocols (ChaCha20, Curve25519)

Rejection Reasons:

• WireGuard is node-level tunneling; does not distinguish services/processes; cannot implement workload-level identity
• Cannot integrate with SPIFFE identity framework; weak tenant isolation
• Certificate/key rotation requires manual configuration or additional tools; low automation
• Still based on “node trust,” not true Zero Trust

Performance Impact Assessment

mTLS performance impact measured data (standard test environment: 4 cores / 8GB / Ubuntu 22.04):

Key optimization measures:

1. TLS 1.3 Session Resumption: Reuse established TLS sessions, avoiding full handshake
2. Connection pool: Long-term keep-alive connections, avoiding frequent handshakes
3. ECDSA-P256: Compared to RSA, signature verification 3-5x faster
4. AES-NI hardware acceleration: Leverage CPU instruction set to accelerate symmetric encryption

Implementation Path

Remediation executed in four phases, totaling 8 weeks:

Phase 1 (weeks 1-2): CA and identity infrastructure
  ├─ Implement built-in CA (Raft-based high-availability cluster)
  ├─ Integrate SPIFFE identity framework
  ├─ Implement HSM adapter layer
  └─ Complete unit testing and security audit

Phase 2 (weeks 3-4): mTLS integration
  ├─ Implement mTLS server in eneros-gateway
  ├─ Implement mTLS client in eneros-agent
  ├─ Convert all internal gRPC to mTLS
  └─ Enforce switch: cannot disable mTLS

Phase 3 (weeks 5-6): Audit and authorization
  ├─ Implement chained audit log
  ├─ Deploy WORM storage
  ├─ Implement RBAC + ABAC authorization engine
  └─ Integrate into all system calls

Phase 4 (weeks 7-8): Testing and acceptance
  ├─ MLPS 2.0 Level 3 re-testing
  ├─ Performance benchmark testing
  ├─ Fault injection (CA downtime, certificate expiry)
  └─ Trial run and operations training

Progressive Enablement Strategy

To reduce risk, mTLS uses progressive enablement in the pilot environment:

# Phase 1: Audit mode only (not enforced, only logs unencrypted connections)
[security.mtls]
mode = "audit"          # audit / permissive / enforce

# Phase 2: Permissive mode (allows unencrypted but alerts)
[security.mtls]
mode = "permissive"

# Phase 3: Enforce mode (rejects unencrypted connections)
[security.mtls]
mode = "enforce"

Corresponding code implementation:

pub enum MtlsMode {
    Audit,       // Record only, do not block
    Permissive,  // Alert, but do not block
    Enforce,     // Enforce, reject unencrypted
}

impl MtlsEnforcer {
    pub fn check(&self, conn: &Connection) -> Result<()> {
        match self.mode {
            MtlsMode::Audit => {
                if !conn.is_mtls() {
                    self.audit_log.record("unencrypted connection", conn);
                }
                Ok(())
            }
            MtlsMode::Permissive => {
                if !conn.is_mtls() {
                    self.metrics.increment("unencrypted_connection_warning");
                    tracing::warn!("unencrypted connection from {:?}", conn.peer);
                }
                Ok(())
            }
            MtlsMode::Enforce => {
                if !conn.is_mtls() {
                    return Err(SecurityError::UnencryptedConnectionRejected);
                }
                Ok(())
            }
        }
    }
}

References

• ADR Overview — ADR index and writing guidelines
• ADR-0002 Power-Native AgentOS — Power-Native design decision
• Zero Trust & Security Enhancement — Capability document
• Zero Trust Architecture & mTLS compliance mapping — Compliance document
• NIST SP 800-207, “Zero Trust Architecture”
• IEC 62443-3-3, “System security requirements and security levels”
• NERC CIP-005, “Electronic Security Perimeter(s)”
• SPIFFE specification, https://spiffe.io
• RFC 8446, “The Transport Layer Security (TLS) Protocol Version 1.3”
• Related crates: eneros-trust, eneros-gateway, eneros-audit, eneros-ids