arXiv subjects

Lukas Wesemann

Publications and source records attributed to Lukas Wesemann.

14 recordsLinked to original sources

arXiv2026

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

Apple's M5 generation introduces a redesigned GPU architecture in which every core carries a dedicated Neural Accelerator: on-die matrix units exposed through the Metal~4 tensor API. We show that BaseRT, our native Metal inference runtime for large language models on Apple Silicon, exploits these units to push inference throughput on Apple hardware substantially beyond both llama.cpp and MLX. Building on BaseRT's framework-free design, we add a family of hand-written Metal~4 tensor-core kernels (including dense and mixture-of-experts GEMM and flash-attention prefill kernels) that route the compute-bound matrix multiplications of inference through the M5 Neural Accelerators while leaving the memory-bound decode path on our existing specialised kernels. On an Apple M5 Pro, across fifteen model configurations spanning the Qwen3, Qwen3.5/3.6, Llama~3.2, and Gemma~4 families from sub-1B to 35B parameters, BaseRT delivers up to $6.4\times$ higher prompt-processing throughput than llama.cpp and $3.9\times$ higher than MLX, with the largest margins on the mixture-of-experts models where matrix multiplication dominates, while maintaining its lead on decode of up to $1.75\times$ over llama.cpp and $1.33\times$ over MLX. These results establish a new performance ceiling for on-device LLM inference and show that the M5's tensor cores are the decisive lever for prompt processing on Apple Silicon. BaseRT is publicly available at https://github.com/basecompute/baseRT.

cs.AR↗

arXiv2026

BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

We present BaseRT, a native Metal inference runtime for large language models (LLMs) on Apple Silicon, and report the highest inference throughput on this hardware to date. Existing runtimes, including llama.cpp and MLX-based frameworks, incur overhead from abstractions not designed for Metal's execution model or Apple Silicon's unified memory topology. By building natively on Metal with chip-specific kernel fusion, unified memory-aware optimisation, and custom dispatch logic, BaseRT recovers performance that framework-based approaches leave on the table. BaseRT supports a wide range of model families across eight quantisation formats (Q2 to FP16) on all Apple M-series devices. In this paper, we evaluate the Qwen3, Llama 3.2, and Gemma 4 families at Q4 and Q8 quantisation on M3 and M4 Pro devices. BaseRT achieves up to 1.56x higher decode throughput than llama.cpp and up to 1.35x higher than MLX, with substantially larger margins on prefill for mixture-of-experts models, delivering consistent best-in-class throughput from sub-1B to 30B parameter models. These results establish Apple Silicon as a more capable inference platform than previously reported, with direct implications for the emerging edge inference paradigm: as privacy requirements, latency constraints, and cloud cost pressures drive inference toward on-device deployment, performance-optimised local runtimes are a critical enabling layer for this transition. BaseRT is publicly available at https://github.com/basecompute/baseRT

cs.CL↗

arXiv2026

Multifunctional Imaging with an Inverse-Designed Nonlocal Metasurface

Nonlocal metasurfaces enable all-optical processing of spatial information in optical fields. Here, we demonstrate a topology-optimised metasurface that switches between phase-contrast and brightfield imaging modalities via polarisation control, eliminating the need to reposition optical components or use computational techniques to image transparent samples. Specifically, for one polarisation state, an asymmetric transfer function about normal incidence performs a first order derivative on the phase, producing pseudo-3D phase-contrast images of transparent biological samples while the orthogonal state returns the result of the identity operator. This work extends inverse-design methods to reconfigurable phase-contrast microscopy and quantitative analogue optical computation with applications in biological imaging, medical diagnostics, and materials characterisation.

physics.optics↗

arXiv2026

More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

Reasoning has emerged as a pivotal capability in Large Language Models (LLMs). Through Reinforcement Learning (RL), typically Group Relative Policy Optimization (GRPO), these models are able to solve complex tasks such as mathematics and code generation. Building on these advances, recent research has sought to extend reasoning to Vision-Language Models (VLMs), yielding promising results across diverse visual tasks. Despite this progress, our study uncovers the dual nature of multimodal reasoning: while it substantially enhances logical inference and facilitates performance on challenging problems, it may gradually impair perceptual grounding, leading to recognition failures on otherwise basic visual questions. Through further analysis, we attribute this phenomenon to visual forgetting, wherein prolonged reasoning causes the model to increasingly disregard visual input. To address this, we propose Vision-Anchored Policy Optimization (VAPO), a simple yet effective method that explicitly steers the reasoning process toward visually grounded trajectories. Our result model, VAPO-Thinker-7B, significantly strengthens the model's reliance on visual information and achieves new state-of-the-art results on a wide range of established benchmarks. Project page: https://xytian1008.github.io/VAPO/

cs.CV↗

arXiv2026

Wavelength and Polarization Multiplexed Nonlocal Metasurface for Quantitative Phase Microscopy

Imaging transparent samples remains an ongoing challenge in the study of unstained biological cells and material samples. Widely used methods trade off system complexity, cost and bulk, computational efficiency and information content. Here we demonstrate the use of a nonlocal metasurface located in the object plane for obtaining single-shot, low-noise differential phase contrast images visualising phase gradients along orthogonal directions in a sample obtained at wavelengths of 613 nm and 656 nm. Furthermore, we show that these images are sufficient to calculate the quantitative phase introduced into the transmitted optical field by the sample. We find that the recovered phase of an optical field generated by a spatial light modulator is in good agreement with specified values. We also present information-rich differential phase contrast images of unstained HeLa cells with the recovered phase excursion values consistent with the literature. Our results demonstrate the potential for metasurfaces as a platform for extracting information from an optical field for use in next-generation compact imaging systems with applications in medical diagnostics, biotechnology, and materials science.

physics.optics↗

arXiv2025

Quantum Phase Gradient Imaging Using a Nonlocal Metasurface System

Quantum phase imaging enables the analysis of transparent samples with thickness and refractive index variations in scenarios requiring precise measurements under low-light conditions. Here, we present a compact quantum phase-gradient imaging system integrating a lithium niobate (LiNbO3) metasurface for generating spatially entangled photon pairs and a silicon (Si) metasurface for phase gradient extraction. By leveraging nonlocal resonances, the LiNbO3 metasurface enables efficient spontaneous parametric down-conversion (SPDC) with all-optically angularly tunable emission, while the Si metasurface provides a nearly linear optical transfer function (OTF) that differentiates the photon wavefunction and extracts phase gradients.Experimental proof-of-concept results demonstrate the imaging of up to 25~rad/mm phase gradients, achieving 89% similarity with the reference values. The pixel resolution of the system can be potentially enhanced by orders of magnitude by increasing the metasurface dimensions and resonance quality factor.Our work showcases the application of metasurfaces in both generating and detecting quantum states and establishes a new paradigm for portable quantum phase-gradient imaging, with potential applications in quantum sensing, microscopy, and LiDAR technology.

physics.optics↗

arXiv2025

Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting

Prompting has emerged as a practical way to adapt frozen vision-language models (VLMs) for video anomaly detection (VAD). Yet, existing prompts are often overly abstract, overlooking the fine-grained human-object interactions or action semantics that define complex anomalies in surveillance videos. We propose ASK-Hint, a structured prompting framework that leverages action-centric knowledge to elicit more accurate and interpretable reasoning from frozen VLMs. Our approach organizes prompts into semantically coherent groups (e.g. violence, property crimes, public safety) and formulates fine-grained guiding questions that align model predictions with discriminative visual cues. Extensive experiments on UCF-Crime and XD-Violence show that ASK-Hint consistently improves AUC over prior baselines, achieving state-of-the-art performance compared to both fine-tuned and training-free methods. Beyond accuracy, our framework provides interpretable reasoning traces towards anomaly and demonstrates strong generalization across datasets and VLM backbones. These results highlight the critical role of prompt granularity and establish ASK-Hint as a new training-free and generalizable solution for explainable video anomaly detection.

cs.CV↗

arXiv2025

Robust Spatiotemporal Forecasting Using Adaptive Deep-Unfolded Variational Mode Decomposition

Accurate spatiotemporal forecasting is critical for numerous complex systems but remains challenging due to complex volatility patterns and spectral entanglement in conventional graph neural networks (GNNs). While decomposition-integrated approaches like variational mode graph convolutional network (VMGCN) improve accuracy through signal decomposition, they suffer from computational inefficiency and manual hyperparameter tuning. To address these limitations, we propose the mode adaptive graph network (MAGN) that transforms iterative variational mode decomposition (VMD) into a trainable neural module. Our key innovations include (1) an unfolded VMD (UVMD) module that replaces iterative optimization with a fixed-depth network to reduce the decomposition time (by 250x for the LargeST benchmark), and (2) mode-specific learnable bandwidth constraints (αk ) adapt spatial heterogeneity and eliminate manual tuning while preventing spectral overlap. Evaluated on the LargeST benchmark (6,902 sensors, 241M observations), MAGN achieves an 85-95% reduction in the prediction error over VMGCN and outperforms state-of-the-art baselines.

cs.LG↗

arXiv2025

Variational Mode-Driven Graph Convolutional Network for Spatiotemporal Traffic Forecasting

This paper focuses on spatiotemporal (ST) traffic prediction using graph neural networks (GNNs). Given that ST data comprises non-stationary and complex temporal patterns, interpreting and predicting such trends is inherently challenging. Representing ST data in decomposed modes helps infer underlying behavior and assess the impact of noise on predictive performance. We propose a framework that decomposes ST data into interpretable modes using variational mode decomposition (VMD) and processes them through a neural network for future state forecasting. Unlike existing graph-based traffic forecasters that operate directly on raw or aggregated time series, the proposed hybrid approach, termed the Variational Mode Graph Convolutional Network (VMGCN), first decomposes non-stationary signals into interpretable variational modes by determining the optimal mode count via reconstruction-loss minimization and then learns both intramode and cross-mode spatiotemporal dependencies through a novel attention-augmented GCN. Additionally, we analyze the significance of each mode and the effect of bandwidth constraints on multi-horizon traffic flow predictions. The proposed two-stage design yields significant accuracy gains while providing frequency-level interpretability with demonstrated superior performance on the LargeST dataset for both short-term and long-term forecasting tasks. The implementation is publicly available on https://github.com/OsamaAhmad369/VMGCN.

cs.LG↗

arXiv2024

Metasurfaces for infrared multi-modal microscopy: phase contrast and bright field

Different imaging modalities are used to extract the diverse information carried in an optical field. Two prominent modalities include bright field and phase contrast microscopy that can visualize the amplitude and phase features of a sample, respectively. However, capturing both of these images on the same camera typically requires interchanging optical components. Metasurfaces are ultra-thin nanostructures that can merge both of these operations into a single miniaturized device. Here, a silicon-based metasurface that supports a Mie resonance is demonstrated to perform near-infrared phase contrast and bright field multi-modal microscopy that can be tuned by changing the polarization of the illumination. We performed experiments using optical fields with phase variations synthesized by a spatial light modulator and introduced by propagation through semi-transparent samples, including C. elegans, unstained human prostate cancer cells and breast tissue. The results demonstrate the potential of metasurfaces for label-free point-of-care testing.

physics.optics↗

arXiv2023

Reconfigurable Image Processing Metasurfaces with Phase-Change Materials

Optical metasurfaces have been enabling reduced footprint and power consumption, as well as faster speeds, in the context of analog computing and image processing. While various image processing and optical computing functionalities have been recently demonstrated using metasurfaces, most of the considered devices are static and lack reconfigurability. Yet, the ability to dynamically reconfigure processing operations is key for metasurfaces to be able to compete with practical computing systems. Here, we demonstrate a passive edge-detection metasurface operating in the near-infrared regime whose image processing response can be drastically modified by temperature variations smaller than 10° C around a CMOS-compatible temperature of 65° C. Such reconfigurability is achieved by leveraging the insulator-to-metal phase transition of a thin buried layer of vanadium dioxide which, in turn, strongly alters the nonlocal response of the metasurface. Importantly, this reconfigurability is accompanied by performance metrics - such as high numerical aperture, high efficiency, isotropy, and polarization-independence - close to optimal, and it is combined with a simple geometry compatible with large-scale manufacturing. Our work paves the way to a new generation of ultra-compact, tunable, passive devices for all-optical computation, with potential applications in augmented reality, remote sensing and bio-medical imaging.

physics.optics↗

arXiv2022

Thin film notch filters as platforms for biological image processing

Many image processing operations involve the modification of the spatial frequency content of images. Here we demonstrate object-plane spatial frequency filtering utilizing the angular sensitivity of a commercial spectral bandstop filter. This approach to all-optical image processing is shown to generate real-time pseudo-3D images of transparent biological and other samples, such as human cervical cancer cells. This work demonstrates the potential of non-local, non-interferometric approaches to image processing for uses in label-free biological cell imaging and dynamical monitoring.

physics.optics↗

arXiv2021

Real time phase imaging with an asymmetric transfer function metasurface

The conversion of phase variations in an optical wavefield into intensity information is of fundamental importance for optical imaging technology including microscopy of biological cells. While conventional approaches to phase-imaging commonly rely on bulky optical components or computational post processing, meta-optical devices have recently demonstrated all-optical, ultracompact image processing methods. Here we describe a metasurface that exploits photonic spin-orbit coupling to create an asymmetric optical transfer function for real time phase-imaging. The effect of the asymmetry on transmission through the device is demonstrated experimentally with the generation of high contrast pseudo-3D intensity images of phase variations in an optical wavefield without the need for post-processing. This non-interferometric method has potential applications in biological live cell imaging and real-time wavefront sensing.

physics.optics↗

arXiv2019

Near-perfect absorber as a subwavelength thickness spatial frequency filter for optical image processing

Spatial frequency filtering is a fundamental enabler of information processing methods in biological and technical imaging. Most filtering methods, however, require either bulky and expensive optical equipment or some degree of computational processing. Here we experimentally demonstrate on-chip, all-optical spatial frequency filtering using a thin-film perfect absorber structure. We give examples of edge enhancement in an amplitude image as well as conversion of a phase gradient in a wave field into an intensity modulation.

physics.optics↗