HE2C: A Holistic Approach for Allocating Latency-Sensitive AI Tasks across Edge-Cloud

Abstract

The high computational, memory, and energy demands of Deep Learning (DL) applications often exceed the capabilities of battery-powered edge devices, creating difficulties in meeting task deadlines and accuracy requirements. Unlike previous solutions that optimize a single metric (e.g., accuracy or energy efficiency), HE2C framework is designed to holistically address the latency, memory, accuracy, throughput, and energy demands of DL applications across edge-cloud continuum, thereby, delivering a more comprehensive and effective user experience. HE2C comprises three key modules: (a) a "feasibility-check module that evaluates the likelihood of meeting deadlines across both edge and cloud resources; (b) a "resource allocation strategy" that maximizes energy efficiency without sacrificing the inference accuracy; and (c) a "rescue module" that enhances throughput by leveraging approximate computing to trade accuracy for latency when necessary. Our primary objective is to maximize system prolong battery lifespan, throughput, and accuracy while adhering to strict latency constraints. Experimental evaluations in the context of wearable technologies for blind and visually impaired users demonstrate that HE2C significantly improves task throughput via completing a larger number of tasks within their specified deadlines, while preserving edge device battery and maintaining prediction accuracy with minimal latency impact. These results underscore HE2C's potential as a robust solution for resource management in latency-sensitive, energy-constrained edge-to-cloud environments.