Multi-Tenant Deployment in Practice

This tutorial demonstrates how to host multiple power operator tenants in a single EnerOS cluster, achieving physical isolation of topology, data, and quotas.

Multi-Tenant Architecture Overview

EnerOS’s multi-tenant system is provided by the eneros-tenant crate. The core abstraction is a five-tuple of “Tenant — Member — Quota — Usage — API Key binding”. All tenant metadata is persisted in SQLite (SqliteTenantStorage), and upper-layer services perform business operations through two objects: TenantManager and TenantUsageTracker.

Tenant state machine:

Prerequisites

• EnerOS cluster deployed (3+ nodes, refer to HA Cluster Deployment)
• Internal CA and mTLS certificate issuance process completed
• SQLite 3.34+ (cluster uses shared block storage mounted at /var/lib/eneros/tenant.db, or a Raft replication engine proxy)
• eneros-cli installed, with the tenant:admin role

Step 1: Plan Tenants

Assign each tenant a unique tenant_id, an independent network namespace, and a certificate SAN. It is recommended to use a “business level + short code” naming convention to avoid conflicts with UUID v4 primary keys.

Quota tier reference table:

Step 2: Configure Tenant Isolation

Declare tenants in eneros.toml:

# /etc/eneros/eneros.toml
[tenant]
db_path = "/var/lib/eneros/tenant.db"
default_quota = { max_devices = 1000, max_agents = 50, max_api_calls_per_day = 100000, max_storage_mb = 10240 }

[[tenants]]
id = "gs"
display_name = "National Dispatch Center"
network = "gs-net"
database = "eneros_gs"
cert_san = ["*.gs.eneros.internal"]
quota = { max_devices = 10000, max_agents = 200, max_api_calls_per_day = 1000000, max_storage_mb = 102400 }

[[tenants]]
id = "ps"
display_name = "Provincial Dispatch Center"
network = "ps-net"
database = "eneros_ps"
cert_san = ["*.ps.eneros.internal"]
quota = { max_devices = 2000, max_agents = 50, max_api_calls_per_day = 200000, max_storage_mb = 20480 }

[[tenants]]
id = "ds"
display_name = "Regional Dispatch Center"
network = "ds-net"
database = "eneros_ds"
cert_san = ["*.ds.eneros.internal"]
quota = { max_devices = 500, max_agents = 20, max_api_calls_per_day = 50000, max_storage_mb = 5120 }

Field descriptions:

Step 3: Create Tenants with TenantManager

Programmatically create tenants via TenantManager. create_tenant automatically generates a UUID v4 as the primary key, and initializes an all-zero usage row in Active status.

use std::sync::Arc;
use eneros_tenant::{
    SqliteTenantStorage, TenantManager, TenantQuota, TenantStatus,
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Open SQLite storage (auto-creates tables + WAL mode + foreign_keys = ON)
    let storage = Arc::new(SqliteTenantStorage::new("/var/lib/eneros/tenant.db")?);

    // 2. Build the Manager
    let manager = TenantManager::new(storage.clone());

    // 3. Idempotently ensure the default tenant exists (called on first deployment)
    let default = manager.ensure_default_tenant()?;
    println!("default tenant: id={}, status={:?}", default.id, default.status);

    // 4. Create the National Dispatch tenant (L3 quota)
    let gs_quota = TenantQuota {
        max_devices: 10_000,
        max_agents: 200,
        max_api_calls_per_day: 1_000_000,
        max_storage_mb: 102_400,
    };
    let gs = manager.create_tenant("National Dispatch Center", gs_quota)?;
    println!("created tenant: id={}, name={}", gs.id, gs.name);

    // 5. Create the Provincial Dispatch tenant (L2 quota)
    let ps_quota = TenantQuota {
        max_devices: 2_000,
        max_agents: 50,
        max_api_calls_per_day: 200_000,
        max_storage_mb: 20_480,
    };
    let ps = manager.create_tenant("Provincial Dispatch Center", ps_quota)?;
    println!("created tenant: id={}, name={}", ps.id, ps.name);

    // 6. List all active tenants
    let active = manager.list_tenants(false)?;
    println!("active tenants: {}", active.len());
    for t in &active {
        println!("  - {} ({}) quotas={:?}", t.name, t.id, t.quotas);
    }

    Ok(())
}

Example output:

default tenant: id=default, status=Active
created tenant: id=8a7b3c2d-1e2f-4a3b-9c8d-7e6f5a4b3c2d, name=National Dispatch Center
created tenant: id=2d3e4f5a-6b7c-8d9e-0f1a-2b3c4d5e6f7a, name=Provincial Dispatch Center
active tenants: 3
  - Default Tenant (default) quotas=TenantQuota { max_devices: 1000, max_agents: 50, max_api_calls_per_day: 100000, max_storage_mb: 10240 }
  - National Dispatch Center (8a7b3c2d-1e2f-4a3b-9c8d-7e6f5a4b3c2d) quotas=TenantQuota { max_devices: 10000, max_agents: 200, max_api_calls_per_day: 1000000, max_storage_mb: 102400 }
  - Provincial Dispatch Center (2d3e4f5a-6b7c-8d9e-0f1a-2b3c4d5e6f7a) quotas=TenantQuota { max_devices: 2000, max_agents: 50, max_api_calls_per_day: 200000, max_storage_mb: 20480 }

Note: The UUID automatically generated by create_tenant and the short code (e.g. gs) in eneros.toml are two different identifiers. In production, it is recommended to write back the name field with a short code prefix via update_tenant after creation, or explicitly call the storage layer create_tenant to write a custom ID in addition to the fixed ID of the default tenant.

Step 4: Issue Tenant Certificates

Each certificate binds tenant_id in its SAN, enforced during the mTLS handshake. The certificate CN uses the three-segment naming <service>.<tenant>.eneros.internal:

use eneros_security::{CertificateAuthority, CertificateSpec};

async fn issue_tenant_certs(ca: &CertificateAuthority) -> anyhow::Result<()> {
    // National Dispatch Agent Runtime certificate
    let gs_cert = ca.issue(
        CertificateSpec::new("agent-runtime.gs.eneros.internal")
            .tenant("gs")
            .sans(vec![
                "agent-runtime.gs.eneros.internal",
                "10.0.0.11",  // IP SAN
            ])
            .role("agent-runtime")
    ).await?;
    println!("issued gs cert: serial={}", gs_cert.serial);

    // Provincial Dispatch Agent Runtime certificate
    let ps_cert = ca.issue(
        CertificateSpec::new("agent-runtime.ps.eneros.internal")
            .tenant("ps")
            .sans(vec!["agent-runtime.ps.eneros.internal"])
            .role("agent-runtime")
    ).await?;
    println!("issued ps cert: serial={}", ps_cert.serial);

    // Provincial Dispatch SCADA Gateway certificate
    let gw_cert = ca.issue(
        CertificateSpec::new("scada-gw.ps.eneros.internal")
            .tenant("ps")
            .sans(vec!["scada-gw.ps.eneros.internal"])
            .role("scada-gateway")
    ).await?;
    println!("issued ps-gw cert: serial={}", gw_cert.serial);

    Ok(())
}

Certificate field conventions:

During the mTLS handshake, the Gateway extracts the tenant_id OID from the certificate and compares it with the tenant prefix in the request path; any mismatch is rejected directly.

Step 5: Member Management and Role Binding

Each tenant independently maintains its member list, and member roles determine their permissions within the tenant.

use eneros_tenant::TenantManager;

fn setup_members(manager: &TenantManager, gs_id: &str, ps_id: &str) -> anyhow::Result<()> {
    // National Dispatch members
    manager.add_member(gs_id, "alice@gs.com", "admin")?;
    manager.add_member(gs_id, "bob@gs.com", "operator")?;
    manager.add_member(gs_id, "carol@grid-vendor.com", "liaison")?; // Vendor liaison

    // Provincial Dispatch members
    manager.add_member(ps_id, "dave@ps.com", "admin")?;
    manager.add_member(ps_id, "eve@ps.com", "operator")?;
    manager.add_member(ps_id, "frank@ps.com", "viewer")?;

    // Verify member existence
    assert!(manager.is_member(gs_id, "alice@gs.com")?);
    assert!(!manager.is_member(ps_id, "alice@gs.com")?); // Cross-tenant invisible

    // List members
    let gs_members = manager.list_members(gs_id)?;
    println!("gs members ({}):", gs_members.len());
    for m in &gs_members {
        println!("  - {} (role={}, added={})",
            m.user_id, m.role, m.added_at);
    }

    Ok(())
}

Role permission matrix:

Step 6: Quota Management and Usage Tracking

TenantUsageTracker provides atomic counters; all read-modify-write operations are protected by the RwLock write lock to ensure concurrency safety.

use std::sync::Arc;
use eneros_tenant::{
    QuotaResource, SqliteTenantStorage, TenantManager,
    TenantQuota, TenantUsageTracker,
};
use eneros_tenant::usage_tracker::spawn_daily_reset_task;

fn setup_quota_and_tracker(
    manager: &TenantManager,
    storage: Arc<SqliteTenantStorage>,
    gs_id: &str,
) -> anyhow::Result<Arc<TenantUsageTracker>> {
    // 1. Adjust quota (e.g. for scale-up)
    let new_quota = TenantQuota {
        max_devices: 15_000,  // Scale from 10000 to 15000
        max_agents: 300,
        max_api_calls_per_day: 2_000_000,
        max_storage_mb: 204_800,
    };
    manager.update_quota(gs_id, &new_quota)?;
    println!("updated gs quota: {:?}", manager.get_quota(gs_id)?);

    // 2. Create the usage tracker
    let tracker = Arc::new(TenantUsageTracker::new(storage));

    // 3. Start the daily reset task at UTC 00:00
    spawn_daily_reset_task(tracker.clone());

    Ok(tracker)
}

fn register_device(
    tracker: &TenantUsageTracker,
    manager: &TenantManager,
    gs_id: &str,
) -> anyhow::Result<()> {
    // 1. Validate quota first (to avoid rollback after exceeding)
    let quota = manager.get_quota(gs_id)?;
    tracker.check_quota(gs_id, QuotaResource::Devices, &quota)?;

    // 2. Actually register the device (business logic omitted)
    // ...device_registry.add(...)

    // 3. Increment the counter
    let new_count = tracker.increment_device_count(gs_id)?;
    println!("gs current_devices = {}", new_count);

    Ok(())
}

fn handle_api_request(
    tracker: &TenantUsageTracker,
    manager: &TenantManager,
    gs_id: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    // Deduct API call quota for each request
    let quota = manager.get_quota(gs_id)?;
    tracker.check_quota(gs_id, QuotaResource::ApiCalls, &quota)?;
    let calls = tracker.increment_api_calls(gs_id)?;
    println!("gs api_calls_today = {}/{}", calls, quota.max_api_calls_per_day);
    Ok(())
}

TenantUsageTracker main methods:

QuotaExceeded error carries resource / limit / current fields; the Gateway returns HTTP 429 based on this:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 3600

{
  "error": "quota_exceeded",
  "resource": "api_calls",
  "limit": 1000000,
  "current": 1000000,
  "reset_at": "2026-07-07T00:00:00Z"
}

Step 7: API Key Binding and Request Routing

For lightweight clients that do not support mTLS (e.g. mobile apps, web frontends), use API Keys for tenant routing. Each API Key is bound to a unique (tenant_id, username, role) triple.

use eneros_tenant::TenantManager;

fn bind_api_keys(manager: &TenantManager, gs_id: &str, ps_id: &str) -> anyhow::Result<()> {
    // API Key for the National Dispatch admin user (in production, generate via KMS)
    manager.bind_api_key(
        "ek_live_gs_alice_5f8a7b2c9d1e3f4a",
        "alice@gs.com",
        "admin",
        gs_id,
    )?;

    // API Key for the Provincial Dispatch operator user
    manager.bind_api_key(
        "ek_live_ps_dave_2a3b4c5d6e7f8a9b",
        "dave@ps.com",
        "operator",
        ps_id,
    )?;

    // Verify binding
    let binding = manager.get_api_key_tenant("ek_live_gs_alice_5f8a7b2c9d1e3f4a")?
        .expect("API key should be bound");
    assert_eq!(binding.tenant_id, gs_id);
    assert_eq!(binding.username, "alice@gs.com");
    assert_eq!(binding.role, "admin");

    // Unknown key returns None
    assert!(manager.get_api_key_tenant("ek_live_unknown_xxx")?.is_none());

    Ok(())
}

Gateway middleware example:

use axum::{extract::Request, http::StatusCode, middleware::Next, response::Response};
use eneros_tenant::TenantManager;

pub async fn tenant_routing_middleware(
    manager: TenantManager,
    req: Request,
    next: Next,
) -> Result<Response, StatusCode> {
    // 1. Prefer extracting tenant_id from the mTLS certificate
    let tenant_id = if let Some(cert) = req.extensions().get::<ClientCert>() {
        cert.tenant_id.clone()
    } else {
        // 2. Fall back to API Key
        let api_key = req
            .headers()
            .get("X-API-Key")
            .and_then(|h| h.to_str().ok())
            .ok_or(StatusCode::UNAUTHORIZED)?;

        let binding = manager
            .get_api_key_tenant(api_key)
            .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?
            .ok_or(StatusCode::UNAUTHORIZED)?;

        binding.tenant_id
    };

    // 3. Validate tenant status (only Active is allowed)
    manager.validate_tenant_access(&tenant_id)
        .map_err(|e| match e {
            eneros_tenant::TenantError::NotFound(_) => StatusCode::NOT_FOUND,
            eneros_tenant::TenantError::InvalidStatus(_) => StatusCode::FORBIDDEN,
            _ => StatusCode::INTERNAL_SERVER_ERROR,
        })?;

    // 4. Inject tenant context
    req.extensions_mut().insert(TenantContext {
        tenant_id: tenant_id.clone(),
        user_id: binding.username.clone(),
        role: binding.role.clone(),
    });

    Ok(next.run(req).await)
}

Important: get_api_key_tenant only returns bindings for tenants with status='active'. API Keys for Suspended / Deleted tenants are automatically invalidated without manual cleanup.

Step 8: Cross-Tenant Access Control

Cross-tenant access is denied by default; only explicit authorization allows it. The liaison role is used for vendor liaison cross-tenant read-only access.

use eneros_auth::{Policy, Subject, Action, Resource, AuthContext};

async fn setup_cross_tenant_policy(policy: &Policy) -> anyhow::Result<()> {
    // 1. Deny all cross-tenant access by default
    policy.deny(
        Subject::any_tenant(),
        Action::any(),
        Resource::tenant_pattern("*"),
    ).await?;

    // 2. Allow intra-tenant access
    policy.allow(
        Subject::tenant("gs"),
        Action::any(),
        Resource::tenant("gs"),
    ).await?;

    // 3. Vendor liaison read-only cross-tenant access (with conditions)
    policy.allow(
        Subject::tenant("ps"),
        Action::Read,
        Resource::tenant("gs").topic("/topics/state_estimate"),
    )
    .when(|ctx: &AuthContext| ctx.caller.has_role("liaison"))
    .await?;

    // 4. Audit scenario: compliance system cross-tenant read-only
    policy.allow(
        Subject::service("audit-collector"),
        Action::Read,
        Resource::tenant_pattern("*").topic("/topics/audit_log"),
    )
    .with_expiry(chrono::Duration::days(90))
    .await?;

    Ok(())
}

Access decision table:

Step 9: Daily API Counter Reset

The api_calls_today counter is automatically zeroed at UTC 00:00 each day, and last_reset_date is refreshed to the current day. spawn_daily_reset_task sleeps in a background tokio task in a loop until the next UTC midnight.

use std::sync::Arc;
use eneros_tenant::{SqliteTenantStorage, TenantUsageTracker};
use eneros_tenant::usage_tracker::spawn_daily_reset_task;

fn start_daily_reset(storage: Arc<SqliteTenantStorage>) -> Arc<TenantUsageTracker> {
    let tracker = Arc::new(TenantUsageTracker::new(storage));

    // Background task: reset daily at UTC 00:00
    spawn_daily_reset_task(tracker.clone());

    // Immediately check once on startup (to handle day rollover that occurred during downtime)
    if let Err(e) = tracker.maybe_reset_daily() {
        tracing::error!("initial daily reset check failed: {}", e);
    }

    tracker
}

Differences between maybe_reset_daily and reset_daily_counters:

Device count / Agent count / storage usage are not affected by the daily reset; only api_calls_today is zeroed.

Step 10: Complete Deployment Flow

Below is a complete multi-tenant deployment example, covering the entire process from storage initialization to middleware mounting:

use std::sync::Arc;
use axum::{routing::get, Router};
use eneros_tenant::{
    QuotaResource, SqliteTenantStorage, TenantManager, TenantQuota,
    TenantUsageTracker,
};
use eneros_tenant::usage_tracker::spawn_daily_reset_task;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Initialize storage
    let storage = Arc::new(SqliteTenantStorage::new("/var/lib/eneros/tenant.db")?);
    let manager = TenantManager::new(storage.clone());
    let tracker = Arc::new(TenantUsageTracker::new(storage.clone()));

    // 2. Start the daily reset task
    tracker.maybe_reset_daily()?;
    spawn_daily_reset_task(tracker.clone());

    // 3. Idempotently create tenants
    let gs_quota = TenantQuota {
        max_devices: 10_000,
        max_agents: 200,
        max_api_calls_per_day: 1_000_000,
        max_storage_mb: 102_400,
    };
    let gs = manager.ensure_tenant_by_name("National Dispatch Center", gs_quota.clone())?;
    let ps = manager.ensure_tenant_by_name("Provincial Dispatch Center", TenantQuota {
        max_devices: 2_000,
        max_agents: 50,
        max_api_calls_per_day: 200_000,
        max_storage_mb: 20_480,
    })?;

    // 4. Register members
    manager.add_member(&gs.id, "alice@gs.com", "admin")?;
    manager.add_member(&ps.id, "dave@ps.com", "operator")?;

    // 5. Bind API Key
    manager.bind_api_key(
        "ek_live_gs_alice_xxx",
        "alice@gs.com",
        "admin",
        &gs.id,
    )?;

    // 6. Mount the Axum middleware
    let app = Router::new()
        .route("/healthz", get(|| async { "ok" }))
        .route("/api/v1/devices", get(list_devices))
        .layer(axum::middleware::from_fn_with_state(
            manager.clone(),
            tenant_routing_middleware,
        ))
        .with_state(manager);

    // 7. Start the HTTP service
    let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await?;
    tracing::info!("gateway listening on :8080");
    axum::serve(listener, app).await?;

    Ok(())
}

// Business handler reads the current tenant via TenantContext
async fn list_devices(
    ctx: axum::extract::Extension<TenantContext>,
) -> Result<String, StatusCode> {
    let tenant_id = &ctx.0.tenant_id;
    Ok(format!("listing devices for tenant {}", tenant_id))
}

The ensure_tenant_by_name in the above example is a convenience method packaged by the project; internally it looks up by name and calls create_tenant on miss. The original TenantManager only provides ensure_default_tenant.

Validation

Validate whether multi-tenant isolation is in effect item by item against the following checklist:

Concurrency safety validation script:

#[cfg(test)]
mod tests {
    use super::*;
    use std::thread;

    #[test]
    fn test_concurrent_api_calls_are_atomic() {
        let storage = Arc::new(SqliteTenantStorage::new_in_memory().unwrap());
        let tracker = Arc::new(TenantUsageTracker::new(storage.clone()));
        let tenant = setup_test_tenant(&storage);

        let mut handles = vec![];
        for _ in 0..100 {
            let t = tracker.clone();
            let id = tenant.id.clone();
            handles.push(thread::spawn(move || {
                t.increment_api_calls(&id).unwrap();
            }));
        }
        for h in handles {
            h.join().unwrap();
        }

        let usage = tracker.get_usage(&tenant.id).unwrap();
        assert_eq!(usage.api_calls_today, 100, "100 concurrent calls should write exactly 100");
    }
}

Debugging and Troubleshooting

View tenant storage internal state:

# View all tenants
sqlite3 /var/lib/eneros/tenant.db \
  "SELECT id, name, status, max_devices, max_agents FROM tenants;"

# View a tenant's usage
sqlite3 /var/lib/eneros/tenant.db \
  "SELECT * FROM tenant_usage WHERE tenant_id='8a7b3c2d-1e2f-4a3b-9c8d-7e6f5a4b3c2d';"

# View all API Key bindings
sqlite3 /var/lib/eneros/tenant.db \
  "SELECT api_key, username, role, tenant_id FROM tenant_api_keys;"

# Manually trigger daily reset (for emergencies)
sqlite3 /var/lib/eneros/tenant.db \
  "UPDATE tenant_usage SET api_calls_today=0, last_reset_date=date('now');"

Data Table Schema Reference

eneros-tenant uses four tables; all timestamps are stored as RFC 3339 strings, and dates as YYYY-MM-DD:

-- Tenant main table
CREATE TABLE tenants (
    id TEXT PRIMARY KEY,                -- UUID v4 or "default"
    name TEXT NOT NULL,
    status TEXT NOT NULL,               -- 'active' / 'suspended' / 'deleted'
    created_at TEXT NOT NULL,           -- RFC 3339
    updated_at TEXT NOT NULL,           -- RFC 3339
    max_devices INTEGER NOT NULL,
    max_agents INTEGER NOT NULL,
    max_api_calls_per_day INTEGER NOT NULL,
    max_storage_mb INTEGER NOT NULL
);

-- Members table (composite primary key)
CREATE TABLE tenant_members (
    tenant_id TEXT NOT NULL,
    user_id TEXT NOT NULL,
    role TEXT NOT NULL,
    added_at TEXT NOT NULL,
    PRIMARY KEY (tenant_id, user_id),
    FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);

-- Usage table (one row per tenant)
CREATE TABLE tenant_usage (
    tenant_id TEXT PRIMARY KEY,
    current_devices INTEGER NOT NULL,
    current_agents INTEGER NOT NULL,
    api_calls_today INTEGER NOT NULL,
    storage_used_mb INTEGER NOT NULL,
    last_reset_date TEXT NOT NULL,      -- 'YYYY-MM-DD'
    FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);

-- API Key binding table
CREATE TABLE tenant_api_keys (
    api_key TEXT PRIMARY KEY,
    username TEXT NOT NULL,
    role TEXT NOT NULL,
    tenant_id TEXT NOT NULL,
    FOREIGN KEY (tenant_id) REFERENCES tenants(id)
);

SQLite initialization parameters (auto-executed by SqliteTenantStorage::new):

PRAGMA journal_mode = WAL;       -- Write-ahead logging, improves concurrent reads
PRAGMA synchronous = NORMAL;     -- Balance performance and persistence
PRAGMA foreign_keys = ON;        -- Enforce foreign key constraints

Next Steps

• Multi-Tenant and Isolation — detailed isolation model
• HA Cluster Deployment — cluster-level isolation
• Plugin Development — multi-tenant plugin signing and distribution