Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2611.03392

TinyML Hits the Memory Wall: A Benchmark of Ten Sub-Watt Edge Devices

Pietro Bonazzi·Luigi Capogrosso·Ahmet Celik·Rosanna Manzo·Marco Giordano·Szymon Ruciński·Jonathan Hungerbuehler·Michele Magno
View original source ↗Read source document ↗Export metadata ↓

Abstract

Neural network inference at the edge must fit tight energy budgets. Existing benchmarks compare few devices, do not report energy on a common basis, and omit the newest microcontrollers with an NPU or the 1 GHz Cortex-M85. Hardware-aware design also ranks networks by MAC count, yet no study has tested whether this proxy holds across accelerators. We benchmark sub-watt edge AI hardware by evaluating four CNNs (MobileNetV3, MCUNetV1, EfficientNet, SqueezeNet) and two tiny transformers (MobileViT-XXS, DeiT-tiny) on ten commercial devices: an in-sensor DSP (Sony IMX500), four MCUs with an NPU (STM32N6, Alif Ensemble E7/E8, GreenWaves GAP9), and five MCUs without an NPU (STM32U5, Apollo510B, ESP32-S3 Xtensa LX7, RA8M2, STM32V8). Four devices are novel (Alif E8, Apollo510B, RA8M2, STM32V8). Measured as energy above idle, the IMX500 (0.13-0.35 mJ) uses 1.3-3x less energy than the STM32N6 (0.31-1.05 mJ) on every model; the GAP9 uses the least on MobileNetV3 and the Alif E7 the least on MCUNetV1 and SqueezeNet. All four use 6-34x less energy than the STM32V8 (1.85-3.98 mJ), and the Alif E7 and IMX500 form the power-performance Pareto front. Latency on all four accelerators scales with memory traffic (bytes of weights and activations, r >= 0.98) and not with MACs, so they are memory-bound, while no MCU without an NPU is. The NPU advantage grows with intensity from 1.4x over a 1 GHz Cortex-M85 on MCUNetV1 to 8.2x on SqueezeNet. MAC count is therefore a misleading cost proxy for sub-watt accelerators.

Explore related subjects

cs.AR

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Pietro Bonazzi, Luigi Capogrosso, Ahmet Celik, Rosanna Manzo, Marco Giordano, Szymon Ruciński, Jonathan Hungerbuehler, Michele Magno. 2026-11-02. TinyML Hits the Memory Wall: A Benchmark of Ten Sub-Watt Edge Devices. https://arxiv.org/abs/2611.03392

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Sustainable operation of research infrastructure for novel computing

Novel compute systems are an emerging research topic, aiming towards building next-generation compute platforms. For these systems to thrive, they need to be provided as research infrastructure to allow acceptance and usage by a large community. By the example of the neuromorphic BrainScaleS-2 system, we showcase the transformation from a laboratory setup to a publicly available research platform. It is embedded into a purpose-built institute, tightly coupling a conventional cluster with novel compute hardware. Our key contribution is the sustainable operation of such a research system: automation, integration, and resource separation enable maintainability, stable operation, and continuity in a fluctuating academic environment. We implement these concepts by adopting established best practices, such as network infrastructure with strong resource separation, 19-inch rack units, the integration of physical and simulated hardware into CI/CD processes, and system monitoring. We evaluate six years of platform operation comprising 3718 production deployments and an average of 198 hardware CI/CD runs per day, with the integration of simulated and physical hardware increasing the full-stack software line coverage from 57% to 79%. We conclude with an explicit collection of lessons learned during the transformation from single-user laboratory setups to a publicly accessible research platform that, since 2023, has served more than 520 individual researchers.

cs.AR↗
arXiv2026

PIMCQG: Co-Designing Billion-Scale Graph-Based Approximate Nearest Neighbor Search for Processing-in-Memory

Graph-based Approximate Nearest Neighbor Search (ANNS) has emerged as the dominant approach for large-scale AI applications, offering superior accuracy-efficiency trade-offs over alternative methods. However, graph-based ANNS is inherently memory-bound due to frequent and irregular graph traversal. This constrains CPU throughput due to limited memory bandwidth, while GPUs lack sufficient HBM capacity for billion-scale indexes. Processing-in-Memory (PIM) offers massive internal bandwidth by co-locating computation with data, making it a natural fit for irregular, memory-bound workloads. However, directly mapping graph-based ANNS onto PIM exposes severe algorithm-architecture mismatches: restricted per-PU local memory, costly inter-PU communication, heavy coordination overhead, and weak compute capability, forcing existing PIM-based ANNS to rely on cluster-based algorithms that plateau at low recall. We propose PIMCQG, an algorithm-hardware co-design framework through three optimizations: (1) a compact index reducing PIM-resident footprint by 14.5x; (2) an asynchronous pipelined scheduler fully utilizing host-PIM bandwidth; and (3) a multiplication-free distance kernel with less than 0.08% recall loss. Evaluations on three billion-scale datasets show up to 20x/17.1x throughput over CPU/GPU baselines, 129x over prior PIM accelerators at high recall, and scale well across multi-node and emerging PIM architectures.

cs.AR↗
arXiv2026

TEMPO: A Tag-Based Framework for Efficient Memory Ordering

Weak-memory processors rely on ordering instructions for correctness, yet conventional implementations often enforce them more conservatively than the memory model requires. This over-enforcement manifests as drain-induced retirement stalls at ordering instructions and conservative squash/replay of speculative loads, suppressing legal executions and reducing throughput. We present TEMPO, a tag-based framework for precise micro-architectural implementation of ordering instructions. TEMPO assigns lightweight ordering tags to instructions and decomposes enforcement across retirement-time predicates and completion-time store ordering, allowing the core to enforce required ordering without conservative retirement serialization. TEMPO eliminates unnecessary retirement serialization at ordering instructions and speculative-load squash/replay. In our evaluation, TEMPO reduces geometric-mean normalized execution cycles by 7.9% on native four-thread workloads and improves geometric-mean IPC by 15.9% on an instrumented SPEC2017 dynamic binary translation (DBT) proxy for cross- ISA execution (e.g., x86-on-Arm), while adding only 262 bytes per core.

cs.AR↗