arXiv subjects

Meng Li

Publications and source records attributed to Meng Li.

At least 235 records · Page 13Linked to original sources

arXiv2024

Structure-preserving parametric finite element methods for simulating axisymmetric solid-state dewetting problems with anisotropic surface energies

Solid-state dewetting (SSD), a widespread phenomenon in solid-solid-vapor system, could be used to describe the accumulation of solid thin films on the substrate. In this work, we consider the sharp interface model for axisymmetric SSD with anisotropic surface energy. By introducing two types of surface energy matrices from the anisotropy functions,we aim to design two structure-preserving algorithms for the axisymmetric SSD. The newly designed schemes are applicable to a broader range of anisotropy functions, and we can theoretically prove their volume conservation and energy stability. In addition, based on a novel weak formulation for the axisymmetric SSD, we further build another two numerical schemes that have good mesh properties. Finally, numerous numerical tests are reported to showcase the accuracy and efficiency of the numerical methods.

math.NA↗

arXiv2024

Structure-preserving finite element methods for computing dynamics of rotating Bose-Einstein condensate

This work is concerned with the construction and analysis of structure-preserving Galerkin methods for computing the dynamics of rotating Bose-Einstein condensate (BEC) based on the Gross-Pitaevskii equation with angular momentum rotation. Due to the presence of the rotation term, constructing finite element methods (FEMs) that preserve both mass and energy remains an unresolved issue, particularly in the context of nonconforming FEMs. Furthermore, in comparison to existing works, we provide a comprehensive convergence analysis, offering a thorough demonstration of the methods' optimal and high-order convergence properties. Finally, extensive numerical results are presented to check the theoretical analysis of the structure-preserving numerical method for rotating BEC, and the quantized vortex lattice's behavior is scrutinized through a series of numerical tests.

math.NA↗

arXiv2024

FastQuery: Communication-efficient Embedding Table Query for Private LLM Inference

With the fast evolution of large language models (LLMs), privacy concerns with user queries arise as they may contain sensitive information. Private inference based on homomorphic encryption (HE) has been proposed to protect user query privacy. However, a private embedding table query has to be formulated as a HE-based matrix-vector multiplication problem and suffers from enormous computation and communication overhead. We observe the overhead mainly comes from the neglect of 1) the one-hot nature of user queries and 2) the robustness of the embedding table to low bit-width quantization noise. Hence, in this paper, we propose a private embedding table query optimization framework, dubbed FastQuery. FastQuery features a communication-aware embedding table quantization algorithm and a one-hot-aware dense packing algorithm to simultaneously reduce both the computation and communication costs. Compared to prior-art HE-based frameworks, e.g., Cheetah, Iron, and Bumblebee, FastQuery achieves more than $4.3\times$, $2.7\times$, $1.3\times$ latency reduction, respectively and more than $75.7\times$, $60.2\times$, $20.2\times$ communication reduction, respectively, on both LLAMA-7B and LLAMA-30B.

cs.CR↗

arXiv2024

Stochastic Multivariate Universal-Radix Finite-State Machine: a Theoretically and Practically Elegant Nonlinear Function Approximator

Nonlinearities are crucial for capturing complex input-output relationships especially in deep neural networks. However, nonlinear functions often incur various hardware and compute overheads. Meanwhile, stochastic computing (SC) has emerged as a promising approach to tackle this challenge by trading output precision for hardware simplicity. To this end, this paper proposes a first-of-its-kind stochastic multivariate universal-radix finite-state machine (SMURF) that harnesses SC for hardware-simplistic multivariate nonlinear function generation at high accuracy. We present the finite-state machine (FSM) architecture for SMURF, as well as analytical derivations of sampling gate coefficients for accurately approximating generic nonlinear functions. Experiments demonstrate the superiority of SMURF, requiring only 16.07% area and 14.45% power consumption of Taylor-series approximation, and merely 2.22% area of look-up table (LUT) schemes.

cs.LG↗

arXiv2024

vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs

IoT devices based on microcontroller units (MCU) provide ultra-low power consumption and ubiquitous computation for near-sensor deep learning models (DNN). However, the memory of MCU is usually 2-3 orders of magnitude smaller than mobile devices, which makes it challenging to map DNNs onto MCUs. Previous work separates memory management and kernel implementation for MCU and relies on coarse-grained memory management techniques such as inplace update to reduce memory consumption. In this paper, we propose to coordinate memory management and kernel optimization for DNN inference on MCUs to enable fine-grained memory management. The key idea is to virtualize the limited memory of MCU as a large memory pool. Each kernel divides the memory pool into kernel-specific segments and handles segment load and store while computing DNN layers. Memory consumption can be reduced because using the fine-grained segment-level memory control, we can overlap the memory footprint of different tensors without the need to materialize them at the same time. Following this idea, we implement \ours{} for DNN inference on MCU. Evaluation for single layers on ARM Cortex-M4 and Cortex-M7 processors shows that \ours{} can reduce from $12.0\%$ to $49.5\%$ RAM usage and from $20.6\%$ to $53.0\%$ energy consumption compared to state-of-the-art work. For full DNN evaluation, \ours{} can reduce the memory bottleneck by $61.5\%$, enabling more models to be deployed on low-end MCUs.

cs.AR↗

arXiv2024

EQO: Exploring Ultra-Efficient Private Inference with Winograd-Based Protocol and Quantization Co-Optimization

Private convolutional neural network (CNN) inference based on secure two-party computation (2PC) suffers from high communication and latency overhead, especially from convolution layers. In this paper, we propose EQO, a quantized 2PC inference framework that jointly optimizes the CNNs and 2PC protocols. EQO features a novel 2PC protocol that combines Winograd transformation with quantization for efficient convolution computation. However, we observe naively combining quantization and Winograd convolution is sub-optimal: Winograd transformations introduce extensive local additions and weight outliers that increase the quantization bit widths and require frequent bit width conversions with non-negligible communication overhead. Therefore, at the protocol level, we propose a series of optimizations for the 2PC inference graph to minimize the communication. At the network level, We develop a sensitivity-based mixed-precision quantization algorithm to optimize network accuracy given communication constraints. We further propose a 2PC-friendly bit re-weighting algorithm to accommodate weight outliers without increasing bit widths. With extensive experiments, EQO demonstrates 11.7x, 3.6x, and 6.3x communication reduction with 1.29%, 1.16%, and 1.29% higher accuracy compared to state-of-the-art frameworks SiRNN, COINN, and CoPriv, respectively.

cs.CR↗

arXiv2024

Temporal-Spatial Manipulation of Bi-Focal Bi-Chromatic Fields for Terahertz Radiations

Mixing the fundamental ($ω$) and the second harmonic (2$ω$) waves in gas phase is a widely employed technique for emitting terahertz (THz) pulses. The THz generation driven by bi-chromatic fields can be described by the photocurrent model, where the THz generation is attributed to free electrons ionized by the $ω$ field, and the 2$ω$ field provides a perturbation to break the symmetry of the asymptotic momentum of free electrons. However, we find that the THz radiation is amplified by one order of magnitude when driven by bi-focal bi-chromatic fields, contradicting the common understanding of the photocurrent model. Meanwhile, present measurements demonstrate that the THz radiation mainly originates from the plasma created by the 2$ω$ pulses instead of the $ω$ pulses. Energy transfer from the 2$ω$ beam to the THz beam during the THz generation has been observed, validating the major contribution of the 2$ω$ beam. Furthermore, the THz bandwidth has been observed to extensively exceed the bandwidth of the pump pulse, not be explained by the photocurrent model as well. These counterintuitive results indicate that undiscovered physical mechanisms are involved in bi-chromatic THz generation in plasma, presenting a significant challenge for understanding strong-field nonlinear optics and simultaneously expanding various applications.

physics.optics↗

arXiv2024

A neuroergonomics model to evaluating nuclear power plants operators' performance under heat stress driven by ECG time-frequency spectrums and fNIRS prefrontal cortex network: a CNN-GAT fusion model

Operators experience complicated physiological and psychological states when exposed to extreme heat stress, which can impair cognitive function and decrease performance significantly, ultimately leading to severe secondary disasters. Therefore, there is an urgent need for a feasible technique to identify their abnormal states to enhance the reliability of human-cybernetics systems. With the advancement of deep learning in physiological modeling, a model for evaluating operators' performance driven by electrocardiogram (ECG) and functional near-infrared spectroscopy (fNIRS) was proposed, demonstrating high ecological validity. The model fused a convolutional neural network (CNN) backbone and a graph attention network (GAT) backbone to extract discriminative features from ECG time-frequency spectrums and fNIRS prefrontal cortex (PFC) network respectively with deeper neuroscience domain knowledge, and eventually achieved 0.90 AUC. Results supported that handcrafted features extracted by specialized neuroscience methods can alleviate overfitting. Inspired by the small-world nature of the brain network, the fNIRS PFC network was organized as an undirected graph and embedded by GAT. It is proven to perform better in information aggregation and delivery compared to a simple non-linear transformation. The model provides a potential neuroergonomics application for evaluating the human state in vital human-cybernetics systems under industry 5.0 scenarios.

cs.HC↗

arXiv2024

Structure-preserving, weighted implicit-explicit schemes for multi-phase incompressible Navier-Stokes/Darcy coupled nonlocal Allen-Cahn model

A multitude of substances exist as mixtures comprising multiple chemical components in the natural world. These substances undergo morphological changes under external influences. the phase field model coupled with fluid flow, the dynamic movement and evolution of the phase interface intricately interact with the fluid motion. This article focuses on the N-component models that couple the conservative Allen-Cahn equation with two types of incompressible fluid flow systems: the Navier-Stokes equation and the Darcy equation. By utilizing the scalar auxiliary variable method and the projection method, we innovatively construct two types of structure-preserving weighted implicit-explicit schemes for the coupled models, resulting in fully decoupled linear systems and second-order accuracy in time. The schemes are proved to be mass-conservative. In addition, with the application of $G$-norm inspired by the idea of $G$-stability, we rigorously establish its unconditional energy stability. Finally, the performance of the proposed scheme is verified by some numerical simulations.

math.NA↗

arXiv2024

Ubiquitous short-range order in multi-principal element alloys

Recent research in multi-principal element alloys (MPEAs) has increasingly focused on the exploration and exploitation of short-range order (SRO) to enhance material performance. However, the understanding of SRO formation and the precise tuning of it within MPEAs remains poorly understood, limiting the comprehension of its impact on material properties and impeding the advancement of SRO engineering. Here, leveraging advanced additive manufacturing techniques that produce samples with a wide range of cooling rates (up to 10^7 K/s) and an improved quantitative electron microscopy method, we characterize SRO in three CoCrNi-based MPEAs to unravel the role of processing route and thermal history on SRO. Surprisingly, irrespective of the processing and thermal treatment applied, all samples exhibit similar levels of SRO, suggesting that prevalent SRO may form during the solidification process. Atomistic simulations of solidification verify that local chemical ordering arises in the liquid-solid interface (solidification front) even under the extreme cooling rate of 10^11 K/s. This phenomenon stems from the swift atomic diffusion in the supercooled liquid, which matches or even surpasses the rate of solidification. Therefore, SRO is an inherent characteristic of most MPEAs, insensitive to variations in cooling rates and annealing treatments typically available in experiments. Integrating thermal treatment with other strategies, such as mechanical deformation and irradiation, might be more effective approaches for harnessing SRO to achieve controlled material properties.

cond-mat.mtrl-sci↗

arXiv2024

ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding

Recent advancements in generative large language models (LLMs) have significantly boosted the performance in natural language processing tasks. However, their efficiency is hampered by the inherent limitations in autoregressive token generation. While parallel decoding with token tree verification, e.g., Medusa, has been proposed to improve decoding parallelism and efficiency, it often struggles with maintaining contextual relationships due to its independent token prediction approach and incurs significant verification overhead, especially with large tree sizes and batch processing. In this paper, we propose ProPD, an efficient LLM parallel decoding framework based on dynamic token tree pruning and generation. ProPD features an advanced early pruning mechanism to efficiently eliminate unpromising token sequences to improve verification efficiency. Additionally, it introduces a dynamic token tree generation algorithm to balance the computation and parallelism of the verification phase in real-time and maximize the overall efficiency across different batch sizes, sequence lengths, and tasks, etc. We verify ProPD across a diverse set of datasets, LLMs, and batch sizes and demonstrate ProPD consistently outperforms existing decoding algorithms by 1.1-3.2x.

cs.LG↗

arXiv2024

ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer

Stochastic computing (SC) has emerged as a promising computing paradigm for neural acceleration. However, how to accelerate the state-of-the-art Vision Transformer (ViT) with SC remains unclear. Unlike convolutional neural networks, ViTs introduce notable compatibility and efficiency challenges because of their nonlinear functions, e.g., softmax and Gaussian Error Linear Units (GELU). In this paper, for the first time, a ViT accelerator based on end-to-end SC, dubbed ASCEND, is proposed. ASCEND co-designs the SC circuits and ViT networks to enable accurate yet efficient acceleration. To overcome the compatibility challenges, ASCEND proposes a novel deterministic SC block for GELU and leverages an SC-friendly iterative approximate algorithm to design an accurate and efficient softmax circuit. To improve inference efficiency, ASCEND develops a two-stage training pipeline to produce accurate low-precision ViTs. With extensive experiments, we show the proposed GELU and softmax blocks achieve 56.3% and 22.6% error reduction compared to existing SC designs, respectively and reduce the area-delay product (ADP) by 5.29x and 12.6x, respectively. Moreover, compared to the baseline low-precision ViTs, ASCEND also achieves significant accuracy improvements on CIFAR10 and CIFAR100.

eess.SY↗

arXiv2024

HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inference

Secure two-party computation with homomorphic encryption (HE) protects data privacy with a formal security guarantee but suffers from high communication overhead. While previous works, e.g., Cheetah, Iron, etc, have proposed efficient HE-based protocols for different neural network (NN) operations, they still assume high precision, e.g., fixed point 37 bit, for the NN operations and ignore NNs' native robustness against quantization error. In this paper, we propose HEQuant, which features low-precision-quantization-aware optimization for the HE-based protocols. We observe the benefit of a naive combination of quantization and HE quickly saturates as bit precision goes down. Hence, to further improve communication efficiency, we propose a series of optimizations, including an intra-coefficient packing algorithm and a quantization-aware tiling algorithm, to simultaneously reduce the number and precision of the transferred data. Compared with prior-art HE-based protocols, e.g., CrypTFlow2, Cheetah, Iron, etc, HEQuant achieves $3.5\sim 23.4\times$ communication reduction and $3.0\sim 9.3\times$ latency reduction. Meanwhile, when compared with prior-art network optimization frameworks, e.g., SENet, SNL, etc, HEQuant also achieves $3.1\sim 3.6\times$ communication reduction.

cs.CR↗

arXiv2024

Efficient yet Accurate End-to-End SC Accelerator Design

Providing end-to-end stochastic computing (SC) neural network acceleration for state-of-the-art (SOTA) models has become an increasingly challenging task, requiring the pursuit of accuracy while maintaining efficiency. It also necessitates flexible support for different types and sizes of operations in models by end-to-end SC circuits. In this paper, we summarize our recent research on end-to-end SC neural network acceleration. We introduce an accurate end-to-end SC accelerator based on a deterministic coding and sorting network. In addition, we propose an SC-friendly model that combines low-precision data paths with high-precision residuals. We introduce approximate computing techniques to optimize SC nonlinear adders and provide some new SC designs for arithmetic operations required by SOTA models. Overall, our approach allows for further significant improvements in circuit efficiency, flexibility, and compatibility through circuit design and model co-optimization. The results demonstrate that the proposed end-to-end SC architecture achieves accurate and efficient neural network acceleration while flexibly accommodating model requirements, showcasing the potential of SC in neural network acceleration.

cs.AR↗

arXiv2024

PathFusion: Path-consistent Lidar-Camera Deep Feature Fusion

Fusing 3D LiDAR features with 2D camera features is a promising technique for enhancing the accuracy of 3D detection, thanks to their complementary physical properties. While most of the existing methods focus on directly fusing camera features with raw LiDAR point clouds or shallow-level 3D features, it is observed that directly combining 2D and 3D features in deeper layers actually leads to a decrease in accuracy due to feature misalignment. The misalignment, which stems from the aggregation of features learned from large receptive fields, becomes increasingly more severe as we delve into deeper layers. In this paper, we propose PathFusion as a solution to enable the alignment of semantically coherent LiDAR-camera deep feature fusion. PathFusion introduces a path consistency loss at multiple stages within the network, encouraging the 2D backbone and its fusion path to transform 2D features in a way that aligns semantically with the transformation of the 3D backbone. This ensures semantic consistency between 2D and 3D features, even in deeper layers, and amplifies the usage of the network's learning capacity. We apply PathFusion to improve a prior-art fusion baseline, Focals Conv, and observe an improvement of over 1.6% in mAP on the nuScenes test split consistently with and without testing-time data augmentations, and moreover, PathFusion also improves KITTI $\text{AP}_{\text{3D}}$ (R11) by about 0.6% on the moderate level.

cs.CV↗

arXiv2024

Iterative PnP and its application in 3D-2D vascular image registration for robot navigation

This paper reports on a new real-time robot-centered 3D-2D vascular image alignment algorithm, which is robust to outliers and can align nonrigid shapes. Few works have managed to achieve both real-time and accurate performance for vascular intervention robots. This work bridges high-accuracy 3D-2D registration techniques and computational efficiency requirements in intervention robot applications. We categorize centerline-based vascular 3D-2D image registration problems as an iterative Perspective-n-Point (PnP) problem and propose to use the Levenberg-Marquardt solver on the Lie manifold. Then, the recently developed Reproducing Kernel Hilbert Space (RKHS) algorithm is introduced to overcome the ``big-to-small'' problem in typical robotic scenarios. Finally, an iterative reweighted least squares is applied to solve RKHS-based formulation efficiently. Experiments indicate that the proposed algorithm processes registration over 50 Hz (rigid) and 20 Hz (nonrigid) and obtains competing registration accuracy similar to other works. Results indicate that our Iterative PnP is suitable for future vascular intervention robot applications.

cs.RO↗

arXiv2023

Minimax programming problems subject to addition-Łukasiewicz fuzzy relational inequalities and their optimal solutions

This article focuses on minimax programming problems subject to addition-Łukasiewicz fuzzy relational inequalities. We first establish two necessary and sufficient conditions that a solution of the fuzzy relational inequalities is a minimal one and explore the existence condition of the unique minimal solution. We also supply an algorithm to search for minimal solutions of the fuzzy relational inequalities starting from a given solution. We then apply minimal solutions of the fuzzy relational inequalities to the minimax programming problems for searching optimal solutions. We provide two algorithms to solve a kind of single variable optimization problems, and obtain the greatest optimal solution. The algorithm for finding minimal solutions of a given solution are also used for searching minimal optimal solutions.

math.GM↗

arXiv2023

Infrared anomalies in ultrathin Ti3C2Tx MXene films

Visible transparent but infrared reflective materials are ideal candidates for both transparent conductive films and low-emissivity glass, which are highly desired in a broad variety of areas such as touchscreens and displays, photovoltaics, smart windows, and antistatic coatings. Ultrathin Ti3C2Tx MXene films are emerging as promising low-emissivity transparent candidates. However, the fundamental IR properties of Ti3C2Tx has not been revealed experimentally due to daunting challenges in the preparation of continuous, large-area, and ultrathin films of optical quality on flat substrates. Herein, we proposed a tape-free transfer method that can help prepare centimeter-size and ultrathin (down to 8 nm) Ti3C2Tx films on diverse optical substrates. Benefitting from this method, the refractive index and permittivity for Ti3C2Tx were successfully measured. Ti3C2Tx films exhibit large in-plane permittivity in the IR region, yielding maximum IR reflectance of 88% for bulk films. Interestingly, three anomalies were found in ultrathin Ti3C2Tx films: strong dispersion in the permittivity, interlayer space-dependent optical properties, and abnormally high IR absorption for a 15-nm-thick film. These anomalies are important guidelines in the design of Ti3C2Tx-based low-emissivity transparent films and other related devices, and may inspire other intriguing applications such as ultrathin IR absorption coatings and tunable IR optical devices.

cond-mat.mtrl-sci↗