LUCAS LIEBE[ 01 // SYSTEMS ][ 02 // KITCHEN ]
[ POTSDAM: --:-- CET ⇄ DAEJEON: --:-- KST ]
DATE: 2026-10⚡ Goodput +42% / 7x @ 100ms SLO vs. Rule-based

CHEIS: Distributed Pipeline Optimization in Real-Time Inference Serving at the Edge (SEC '26)

C++LibTorchTensorRTZeroMQContinual RLNVIDIA JetsonDocker
[ ⬇ DOWNLOAD PDF ]

01 // Overview & Core Contribution

Published at SEC ’26 — The 11th ACM/IEEE Symposium on Edge Computing (Co-First Author with Thanh-Tung Nguyen, alongside Nhat-Quang Tau and Dongman Lee at KAIST).

Real-time Edge Inference Serving (EIS) for Physical AI—such as smart factories and multi-camera surveillance—must execute multi-stage deep learning DAGs under strict latency SLOs (100–250 ms) despite bursty video streams and volatile edge resources. Centralized schedulers react too slowly due to multi-second polling cycles, while independent RL agents lack global visibility and induce downstream pipeline bottlenecks.

CHEIS resolves this responsiveness–coordination tension by decoupling the zero-copy inference data plane from an asynchronous, two-timescale control plane linked via Implicit Coordination.

02 // Hierarchical Control Architecture

  • Performance Contracts (τ_m, C_m): A central Cluster Controller partitions end-to-end pipeline SLOs into per-model latency and memory bounds, preventing greedy local optimization from starving downstream stages.
  • Lightweight Continual RL (iAgent): Each containerized model is wrapped by a non-blocking C++/LibTorch daemon (~5,000 parameters, <20 KB) that tunes batch size, batch timeout, and CPU thread concurrency at 1 s granularity using step-size regularized PPO.
  • Diversity-Driven Experience Buffer (xBF): Retains a fixed window of 200 transitions (<8.4 MB) filtered by Mahalanobis state distance and policy KL divergence—preventing catastrophic forgetting under workload drift without large replay buffers.
  • Event-Driven Uncertainty Signaling (ξ, δ, U): Local agents only trigger cluster-wide re-planning when Compliance Score drops, Resource Slack opens up, or Epistemic Uncertainty signals an out-of-distribution shift.

03 // Key Evaluation Highlights

Evaluated in C++ (TensorRT v10.5, LibTorch v2.5, ZeroMQ) across an 11-node heterogeneous Jetson cluster (AGX Xavier, Xavier NX, Orin Nano) and an RTX 3090 edge server over emulated 5G traces:

  • Throughput & Strict SLOs: Sustains up to 99% goodput at a 250 ms P99 SLO—outperforming rule-based baselines by 42% and achieving 7x higher goodput than SOTA RL baselines (BCEdge) under strict 100 ms deadlines.
  • Edge Efficiency: Consumes <3% of device GPU memory (10x lower than BCEdge), adds ~90 mW average power overhead, and completes on-device policy updates in <1 s.
  • Control-Plane Stability: Cuts global cluster re-schedulings by 27% compared to periodic orchestrators.
  • Code & Artifacts: github.com/tungngreen/PipelineScheduler • Zenodo Archive