arXiv subjects

Lei Su

Publications and source records attributed to Lei Su.

At least 37 records · Page 2Linked to original sources

arXiv2024

Mesoscopic Insights: Orchestrating Multi-scale & Hybrid Architecture for Image Manipulation Localization

The mesoscopic level serves as a bridge between the macroscopic and microscopic worlds, addressing gaps overlooked by both. Image manipulation localization (IML), a crucial technique to pursue truth from fake images, has long relied on low-level (microscopic-level) traces. However, in practice, most tampering aims to deceive the audience by altering image semantics. As a result, manipulation commonly occurs at the object level (macroscopic level), which is equally important as microscopic traces. Therefore, integrating these two levels into the mesoscopic level presents a new perspective for IML research. Inspired by this, our paper explores how to simultaneously construct mesoscopic representations of micro and macro information for IML and introduces the Mesorch architecture to orchestrate both. Specifically, this architecture i) combines Transformers and CNNs in parallel, with Transformers extracting macro information and CNNs capturing micro details, and ii) explores across different scales, assessing micro and macro information seamlessly. Additionally, based on the Mesorch architecture, the paper introduces two baseline models aimed at solving IML tasks through mesoscopic representation. Extensive experiments across four datasets have demonstrated that our models surpass the current state-of-the-art in terms of performance, computational complexity, and robustness.

cs.CV↗

arXiv2024

GCUNet: A GNN-Based Contextual Learning Network for Tertiary Lymphoid Structure Semantic Segmentation in Whole Slide Image

We focus on tertiary lymphoid structure (TLS) semantic segmentation in whole slide image (WSI). Unlike TLS binary segmentation, TLS semantic segmentation identifies boundaries and maturity, which requires integrating contextual information to discover discriminative features. Due to the extensive scale of WSI (e.g., 100,000 \times 100,000 pixels), the segmentation of TLS is usually carried out through a patch-based strategy. However, this prevents the model from accessing information outside of the patches, limiting the performance. To address this issue, we propose GCUNet, a GNN-based contextual learning network for TLS semantic segmentation. Given an image patch (target) to be segmented, GCUNet first progressively aggregates long-range and fine-grained context outside the target. Then, a Detail and Context Fusion block (DCFusion) is designed to integrate the context and detail of the target to predict the segmentation mask. We build four TLS semantic segmentation datasets, called TCGA-COAD, TCGA-LUSC, TCGA-BLCA and INHOUSE-PAAD, and make the former three datasets (comprising 826 WSIs and 15,276 TLSs) publicly available to promote the TLS semantic segmentation. Experiments on these datasets demonstrate the superiority of GCUNet, achieving at least 7.41% improvement in mF1 compared with SOTA.

cs.CV↗

arXiv2024

Comprehensive Performance Evaluation of YOLOv11, YOLOv10, YOLOv9, YOLOv8 and YOLOv5 on Object Detection of Power Equipment

With the rapid development of global industrial production, the demand for reliability in power equipment has been continuously increasing. Ensuring the stability of power system operations requires accurate methods to detect potential faults in power equipment, thereby guaranteeing the normal supply of electrical energy. In this article, the performance of YOLOv5, YOLOv8, YOLOv9, YOLOv10, and the state-of-the-art YOLOv11 methods was comprehensively evaluated for power equipment object detection. Experimental results demonstrate that the mean average precision (mAP) on a public dataset for power equipment was 54.4%, 55.5%, 43.8%, 48.0%, and 57.2%, respectively, with the YOLOv11 achieving the highest detection performance. Moreover, the YOLOv11 outperformed other methods in terms of recall rate and exhibited superior performance in reducing false detections. In conclusion, the findings indicate that the YOLOv11 model provides a reliable and effective solution for power equipment object detection, representing a promising approach to enhancing the operational reliability of power systems.

cs.CV↗

arXiv2024

Topology-aware Preemptive Scheduling for Co-located LLM Workloads

Hosting diverse large language model workloads in a unified resource pool through co-location is cost-effective. For example, long-running chat services generally follow diurnal traffic patterns, which inspire co-location of batch jobs to fulfill resource valleys between successive peaks, and thus to saturate resource allocation in cluster-wide scope. These heterogeneous workloads often have different business priorities, and therefore preemption can be leveraged for resource elasticity. However, workloads often have distinct topology preferences as well. The resources released by lower-priority instances may fail to meet the requirements of high-priority online services which are usually latency-sensitive. The root cause behind such mis-match is a lack of topology awareness of resource scheduler, especially during preemption. To bridge this gap, we develop a fine-grained topology-aware method for preemptive scheduling of hybrid workloads. The method ensures that the resources freed by preempted tasks adhere to the topological affinity needs of high-priority preemptors in a guaranteed or best-effort manner. This dynamic alignment significantly increases the efficiency of preemption and improves overall scheduled performance for LLM workloads by $55\%$.

cs.DC↗

arXiv2024

IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization

A comprehensive benchmark is yet to be established in the Image Manipulation Detection & Localization (IMDL) field. The absence of such a benchmark leads to insufficient and misleading model evaluations, severely undermining the development of this field. However, the scarcity of open-sourced baseline models and inconsistent training and evaluation protocols make conducting rigorous experiments and faithful comparisons among IMDL models challenging. To address these challenges, we introduce IMDL-BenCo, the first comprehensive IMDL benchmark and modular codebase. IMDL-BenCo: i) decomposes the IMDL framework into standardized, reusable components and revises the model construction pipeline, improving coding efficiency and customization flexibility; ii) fully implements or incorporates training code for state-of-the-art models to establish a comprehensive IMDL benchmark; and iii) conducts deep analysis based on the established benchmark and codebase, offering new insights into IMDL model architecture, dataset characteristics, and evaluation standards. Specifically, IMDL-BenCo includes common processing algorithms, 8 state-of-the-art IMDL models (1 of which are reproduced from scratch), 2 sets of standard training and evaluation protocols, 15 GPU-accelerated evaluation metrics, and 3 kinds of robustness evaluation. This benchmark and codebase represent a significant leap forward in calibrating the current progress in the IMDL field and inspiring future breakthroughs. Code is available at: https://github.com/scu-zjz/IMDLBenCo.

cs.CV↗

arXiv2024

INT-FlashAttention: Enabling Flash Attention for INT8 Quantization

As the foundation of large language models (LLMs), self-attention module faces the challenge of quadratic time and memory complexity with respect to sequence length. FlashAttention accelerates attention computation and reduces its memory usage by leveraging the GPU memory hierarchy. A promising research direction is to integrate FlashAttention with quantization methods. This paper introduces INT-FlashAttention, the first INT8 quantization architecture compatible with the forward workflow of FlashAttention, which significantly improves the inference speed of FlashAttention on Ampere GPUs. We implement our INT-FlashAttention prototype with fully INT8 activations and general matrix-multiplication (GEMM) kernels, making it the first attention operator with fully INT8 input. As a general token-level post-training quantization framework, INT-FlashAttention is also compatible with other data formats like INT4, etc. Experimental results show INT-FlashAttention achieves 72% faster inference speed and 82% smaller quantization error compared to standard FlashAttention with FP16 and FP8 data format.

cs.LG↗

arXiv2024

ISO: Overlap of Computation and Communication within Seqenence For LLM Inference

In the realm of Large Language Model (LLM) inference, the inherent structure of transformer models coupled with the multi-GPU tensor parallelism strategy leads to a sequential execution of computation and communication. This results in substantial underutilization of computing resources during the communication phase. To mitigate this inefficiency, various techniques have been developed to optimize the use of computational power throughout the communication process. These strategies primarily involve overlapping matrix computations and communications, as well as interleaving micro-batches across different requests. Nonetheless, these approaches either fall short of achieving ideal overlap or impose certain limitations on their application. To overcome these challenges, this paper introduces a novel strategy for computation-communication overlap that operates at the sequence level. This method not only enhances the degree of overlap but also minimizes the constraints on its applicability. Experimental evaluations conducted using 30b/70b models have demonstrated significant improvements in efficiency. Specifically, the proposed technique has been shown to reduce time consumption by approximately 35% on 4090 GPU and by roughly 15% on A800 GPU during the prefill stage of LLM inference.

cs.DC↗

arXiv2024

Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation

Lossless speculative decoding accelerates target large language model (LLM) inference by employing a lightweight draft model for generating tree-structured candidates, which are subsequently verified in parallel by the target LLM. Currently, effective approaches leverage feature-level rather than token-level autoregression within the draft model to facilitate more straightforward predictions and enhanced knowledge distillation. In this paper, we reassess these approaches and propose FSPAD (Feature Sampling and Partial Alignment Distillation for Lossless Speculative Decoding), which introduces two straightforward and effective components within the existing framework to boost lossless speculative decoding. Firstly, FSPAD utilizes token embeddings to sample features of the target LLM in high-dimensional space before feeding them into the draft model, due to the inherent uncertainty of the features preventing the draft model from obtaining the specific token output by the target LLM. Secondly, FSPAD introduces partial alignment distillation to weaken the draft model's connection between features and logits, aiming to reduce the conflict between feature alignment and logit confidence during training. Our experiments include both greedy and non-greedy decoding on the largest and smallest models from the Vicuna and LLaMA3-Instruct series, as well as tasks in multi-turn conversation, translation, summarization, question answering, mathematical reasoning, and retrieval-augmented generation. The results show that FSPAD outperforms the state-of-the-art method across all the aforementioned tasks and target LLMs.

cs.CL↗

arXiv2024

BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline

The general capabilities of Large Language Models (LLM) highly rely on the composition and selection on extensive pretraining datasets, treated as commercial secrets by several institutions. To mitigate this issue, we open-source the details of a universally applicable data processing pipeline and validate its effectiveness and potential by introducing a competitive LLM baseline. Specifically, the data processing pipeline consists of broad collection to scale up and reweighting to improve quality. We then pretrain a 7B model BaichuanSEED with 3T tokens processed by our pipeline without any deliberate downstream task-related optimization, followed by an easy but effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency and predictability throughout training and achieves comparable performance on comprehensive benchmarks with several commercial advanced large language models, such as Qwen1.5 and Llama3. We also conduct several heuristic experiments to discuss the potential for further optimization of downstream tasks, such as mathematics and coding.

cs.CL↗

arXiv2024

Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding

Large Language Models (LLMs) frequently suffer from inefficiencies, largely attributable to the discord between the requirements of auto-regressive decoding and the architecture of contemporary GPUs. Recently, regressive lightweight speculative decoding has garnered attention for its notable efficiency improvements in text generation tasks. This approach utilizes a lightweight regressive draft model, like a Recurrent Neural Network (RNN) or a single transformer decoder layer, leveraging sequential information to iteratively predict potential tokens. Specifically, RNN draft models are computationally economical but tend to deliver lower accuracy, while attention decoder layer models exhibit the opposite traits. This paper presents Clover-2, an advanced iteration of Clover, an RNN-based draft model designed to achieve comparable accuracy to that of attention decoder layer models while maintaining minimal computational overhead. Clover-2 enhances the model architecture and incorporates knowledge distillation to increase Clover's accuracy and improve overall efficiency. We conducted experiments using the open-source Vicuna 7B and LLaMA3-Instruct 8B models. The results demonstrate that Clover-2 surpasses existing methods across various model architectures, showcasing its efficacy and robustness.

cs.CL↗

arXiv2024

Neural Graph Matching for Video Retrieval in Large-Scale Video-driven E-commerce

With the rapid development of the short video industry, traditional e-commerce has encountered a new paradigm, video-driven e-commerce, which leverages attractive videos for product showcases and provides both video and item services for users. Benefitting from the dynamic and visualized introduction of items,video-driven e-commerce has shown huge potential in stimulating consumer confidence and promoting sales. In this paper, we focus on the video retrieval task, facing the following challenges: (1) Howto handle the heterogeneities among users, items, and videos? (2)How to mine the complementarity between items and videos for better user understanding? In this paper, we first leverage the dual graph to model the co-existing of user-video and user-item interactions in video-driven e-commerce and innovatively reduce user preference understanding to a graph matching problem. To solve it, we further propose a novel bi-level Graph Matching Network(GMN), which mainly consists of node- and preference-level graph matching. Given a user, node-level graph matching aims to match videos and items, while preference-level graph matching aims to match multiple user preferences extracted from both videos and items. Then the proposed GMN can generate and improve user embedding by aggregating matched nodes or preferences from the dual graph in a bi-level manner. Comprehensive experiments show the superiority of the proposed GMN with significant improvements over state-of-the-art approaches (e.g., AUC+1.9% and CTR+7.15%). We have developed it on a well-known video-driven e-commerce platform, serving hundreds of millions of users every day

cs.LG↗

arXiv2024

Gapless symmetry-protected topological phases and generalized deconfined critical points from gauging a finite subgroup

Gauging a finite subgroup of a global symmetry can map conventional phases and phase transitions to unconventional ones. In this work, we study, as a concrete example, an emergent $\mathbb{Z}_2$-gauged system with global symmetry $U(1)$, namely, the $\mathbb{Z}_2$-gauged Bose-Hubbard model both in 1-D and in 2-D. In certain limits, there is an emergent mixed 't Hooft anomaly between the quotient $\tilde{U}(1)$ symmetry and the dual $\hat{\mathbb{Z}}_2$ symmetry. In 1-D, the superfluid phase is mapped to an intrinsically gapless symmetry-protected topological (SPT) phase, as supported by density-matrix renormalization group (DMRG) calculations. In 2-D, the original superfluid-insulator transition becomes a generalized deconfined quantum critical point (DQCP) between a gapless SPT phase, where a SPT order coexists with Goldstone modes, and a $\tilde{U}(1)$-symmetry-enriched topological (SET) phase. We also discuss the stability of these phases and the critical points to small perturbations and their potential experimental realizations. Our work demonstrates that partial gauging is a simple and yet powerful approach in constructing novel phases and quantum criticalities.

cond-mat.str-el↗

arXiv2024

Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge

Large language models (LLMs) suffer from low efficiency as the mismatch between the requirement of auto-regressive decoding and the design of most contemporary GPUs. Specifically, billions to trillions of parameters must be loaded to the GPU cache through its limited memory bandwidth for computation, but only a small batch of tokens is actually computed. Consequently, the GPU spends most of its time on memory transfer instead of computation. Recently, parallel decoding, a type of speculative decoding algorithms, is becoming more popular and has demonstrated impressive efficiency improvement in generation. It introduces extra decoding heads to large models, enabling them to predict multiple subsequent tokens simultaneously and verify these candidate continuations in a single decoding step. However, this approach deviates from the training objective of next token prediction used during pre-training, resulting in a low hit rate for candidate tokens. In this paper, we propose a new speculative decoding algorithm, Clover, which integrates sequential knowledge into the parallel decoding process. This enhancement improves the hit rate of speculators and thus boosts the overall efficiency. Clover transmits the sequential knowledge from pre-speculated tokens via the Regressive Connection, then employs an Attention Decoder to integrate these speculated tokens. Additionally, Clover incorporates an Augmenting Block that modifies the hidden states to better align with the purpose of speculative generation rather than next token prediction. The experiment results demonstrate that Clover outperforms the baseline by up to 91% on Baichuan-Small and 146% on Baichuan-Large, respectively, and exceeds the performance of the previously top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on Baichuan-Large, respectively.

cs.CL↗

arXiv2024

Dynamics and Phases of Nonunitary Floquet Transverse-Field Ising Model

Inspired by current research on measurement-induced quantum phase transitions, we analyze the nonunitary Floquet transverse-field Ising model with complex nearest-neighbor couplings and complex transverse fields. Unlike its unitary counterpart, the model shows a number of steady phases, stable to integrability breaking perturbations. Some phases have robust edge modes and/or spatiotemporal long-range orders in the bulk. The transitions between the phases have extensive entanglement entropy, whose scaling with the system size depends on the number of the real quasiparticle modes in the spectrum at the transition. In particular, the volume law scaling appears on some critical lines, protected by pseudo-Hermiticity. Both the scaling of entanglement entropy in steady states and the evolution after a quench are compatible with the non-Hermitian generalization of the quasiparticle picture of Calabrese and Cardy at least qualitatively.

quant-ph↗

arXiv2023

Expanding the Pressure Frontier in Grüneisen Parameter Measurement: Study of Sodium Chloride

The Grüneisen parameter (γ) is crucial for determining many thermal properties, including the anharmonic effect, thermostatistics, and equation of state (EOS) of materials. However, the isentropic adiabatic compression conditions required to measure the Grüneisen parameter under high pressure are difficult to achieve. Thus, direct experimental Grüneisen parameter data in a wide range of pressures is sparse. In this work, we developed a new device that can apply pressure (up to tens of GPa) with an extremely short time about 0.5 ms, confidently achieving isentropic adiabatic compression. Then, we applied our new technique to sodium chloride and measured its Grüneisen parameter, which conforms to previous theoretical predictions. According to our obtained sodium chloride Grüneisen parameters, the calculated Hugoniot curve of the NaCl B1 phase appears up to 20 GPa and 960 K, which compares very well with the shock compression experiment data by Fritz et al. and other calculation works. Our results suggest that this new method can reliably measure the Grüneisen parameter of even more materials, which is significant for researching the equation of state in substances.

cond-mat.mtrl-sci↗

arXiv2023

Boundary criticality via gauging finite subgroups: a case study on the clock model

Gauging a finite Abelian normal subgroup $Γ$ of a nonanomalous 0-form symmetry $G$ of a theory in $(d+1)$D spacetime can yield an unconventional critical point if the original theory has a continuous transition where $Γ$ is completely spontaneously broken and if $G$ is a nontrivial extension of $G/Γ$ by $Γ$. The gauged theory has symmetry $G/Γ\times \hatΓ^{(d-1)}$, where $\hatΓ^{(d-1)}$ is the $(d-1)$-form dual symmetry of $Γ$, and a 't Hooft anomaly between them. Thus it can be viewed as a boundary of a topological phase protected by $G/Γ\times \hatΓ^{(d-1)}$. The ordinary critical point, upon gauging, is mapped to a deconfined quantum critical point between two ordinary symmetry-breaking phases ($d =1$) or an unconventional quantum critical point between an ordinary symmetry-breaking phase and a topologically ordered phase ($d\ge 2$) associated with $G/Γ$ and $\hatΓ^{(d-1)}$, respectively. Order parameters and disorder parameters, before and after gauging, can be directly related. As a concrete example, we gauge the $\mathbb{Z}_2$ subgroup of $\mathbb{Z}_4$ symmetry of a 4-state clock model on a 1D lattice and a 2D square lattice. Since the symmetry of the clock model contains $D_8$, the dihedral group of order 8, we also analyze the anomaly structure which is similar to that in the compactified $SU(2)$ gauge theory with $θ=π$ in $(3+1)$D and its mixed gauge theory. The general case is also discussed.

cond-mat.str-el↗

arXiv2022

Integrable nonunitary quantum circuits

We show that the integrable Lindblad superoperators found recently can be used to build integrable nonunitary quantum circuits with two-site gates by demonstrating that the $\check{R}$-matrices are completely positive and trace preserving. Using the bond-site transformation, we obtain the corresponding integrable nonunitary quantum circuits with three-site gates. When restricted to the diagonals of the density matrix, these quantum models reduce to integrable classical cellular automata. This approach may pave the way for a systematic construction of integrable nonunitary quantum circuits.

cond-mat.str-el↗

arXiv2021

Discovery of carbon-based strongest and hardest amorphous material

Carbon is likely the most fascinating element of the periodic table because of the diversity of its allotropes stemming from its variable (sp, sp2, and sp3) bonding motifs. Exploration of new forms of carbon has been an eternal theme of contemporary scientific research. Here we report on novel amorphous carbon phases containing high fraction of sp3 bonded atoms recovered after compressing fullerene C60 to previously unexplored high pressure and temperature. The synthesized carbons are the hardest and strongest amorphous materials known to date, capable of scratching diamond crystal and approaching its strength which is evidenced by complimentary mechanical tests. Photoluminescence and absorption spectra of the materials demonstrate they are semiconductors with tunable bandgaps in the range of 1.5-2.2 eV, comparable to that of amorphous silicon. A remarkable combination of the outstanding mechanical and electronic properties makes this class of amorphous carbons an excellent candidate for photovoltaic applications demanding ultrahigh strength and wear resistance.

cond-mat.mtrl-sci↗