arXiv subjects

Yu Wang

Publications and source records attributed to Yu Wang.

At least 739 records · Page 41Linked to original sources

arXiv2024

Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator

Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions, yet their application within the medical field remains insufficiently explored. Previous works mainly focus on the performance of medical knowledge with examinations, which is far from the realistic scenarios, falling short in assessing the abilities of LLMs on clinical tasks. In the quest to enhance the application of Large Language Models (LLMs) in healthcare, this paper introduces the Automated Interactive Evaluation (AIE) framework and the State-Aware Patient Simulator (SAPS), targeting the gap between traditional LLM evaluations and the nuanced demands of clinical practice. Unlike prior methods that rely on static medical knowledge assessments, AIE and SAPS provide a dynamic, realistic platform for assessing LLMs through multi-turn doctor-patient simulations. This approach offers a closer approximation to real clinical scenarios and allows for a detailed analysis of LLM behaviors in response to complex patient interactions. Our extensive experimental validation demonstrates the effectiveness of the AIE framework, with outcomes that align well with human evaluations, underscoring its potential to revolutionize medical LLM testing for improved healthcare delivery.

cs.CL↗

arXiv2024

A Survey on Efficient Inference for Large Language Models

Large Language Models (LLMs) have attracted extensive attention due to their remarkable performance across various tasks. However, the substantial computational and memory requirements of LLM inference pose challenges for deployment in resource-constrained scenarios. Efforts within the field have been directed towards developing techniques aimed at enhancing the efficiency of LLM inference. This paper presents a comprehensive survey of the existing literature on efficient LLM inference. We start by analyzing the primary causes of the inefficient LLM inference, i.e., the large model size, the quadratic-complexity attention operation, and the auto-regressive decoding approach. Then, we introduce a comprehensive taxonomy that organizes the current literature into data-level, model-level, and system-level optimization. Moreover, the paper includes comparative experiments on representative methods within critical sub-fields to provide quantitative insights. Last but not least, we provide some knowledge summary and discuss future research directions.

cs.CL↗

arXiv2024

Model Provenance via Model DNA

Understanding the life cycle of the machine learning (ML) model is an intriguing area of research (e.g., understanding where the model comes from, how it is trained, and how it is used). This paper focuses on a novel problem within this field, namely Model Provenance (MP), which concerns the relationship between a target model and its pre-training model and aims to determine whether a source model serves as the provenance for a target model. This is an important problem that has significant implications for ensuring the security and intellectual property of machine learning models but has not received much attention in the literature. To fill in this gap, we introduce a novel concept of Model DNA which represents the unique characteristics of a machine learning model. We utilize a data-driven and model-driven representation learning method to encode the model's training data and input-output information as a compact and comprehensive representation (i.e., DNA) of the model. Using this model DNA, we develop an efficient framework for model provenance identification, which enables us to identify whether a source model is a pre-training model of a target model. We conduct evaluations on both computer vision and natural language processing tasks using various models, datasets, and scenarios to demonstrate the effectiveness of our approach in accurately identifying model provenance.

cs.LG↗

arXiv2024

Simple Graph Condensation

The burdensome training costs on large-scale graphs have aroused significant interest in graph condensation, which involves tuning Graph Neural Networks (GNNs) on a small condensed graph for use on the large-scale original graph. Existing methods primarily focus on aligning key metrics between the condensed and original graphs, such as gradients, output distribution and trajectories of GNNs, yielding satisfactory performance on downstream tasks. However, these complex metrics necessitate intricate external parameters and can potentially disrupt the optimization process of the condensation graph, making the condensation process highly demanding and unstable. Motivated by the recent success of simplified models across various domains, we propose a simplified approach to metric alignment in graph condensation, aiming to reduce unnecessary complexity inherited from intricate metrics. We introduce the Simple Graph Condensation (SimGC) framework, which aligns the condensed graph with the original graph from the input layer to the prediction layer, guided by a pre-trained Simple Graph Convolution (SGC) model on the original graph. Importantly, SimGC eliminates external parameters and exclusively retains the target condensed graph during the condensation process. This straightforward yet effective strategy achieves a significant speedup of up to 10 times compared to existing graph condensation methods while performing on par with state-of-the-art baselines. Comprehensive experiments conducted on seven benchmark datasets demonstrate the effectiveness of SimGC in prediction accuracy, condensation time, and generalization capability. Our code is available at https://github.com/BangHonor/SimGC.

cs.LG↗

arXiv2024

LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction

Existing methods enhance open-vocabulary object detection by leveraging the robust open-vocabulary recognition capabilities of Vision-Language Models (VLMs), such as CLIP.However, two main challenges emerge:(1) A deficiency in concept representation, where the category names in CLIP's text space lack textual and visual knowledge.(2) An overfitting tendency towards base categories, with the open vocabulary knowledge biased towards base categories during the transfer from VLMs to detectors.To address these challenges, we propose the Language Model Instruction (LaMI) strategy, which leverages the relationships between visual concepts and applies them within a simple yet effective DETR-like detector, termed LaMI-DETR.LaMI utilizes GPT to construct visual concepts and employs T5 to investigate visual similarities across categories.These inter-category relationships refine concept representation and avoid overfitting to base categories.Comprehensive experiments validate our approach's superior performance over existing methods in the same rigorous setting without reliance on external training resources.LaMI-DETR achieves a rare box AP of 43.4 on OV-LVIS, surpassing the previous best by 7.8 rare box AP.

cs.CV↗

arXiv2024

Reconstruct the Pruned Model without Any Retraining

Structured pruning is a promising hardware-friendly compression technique for large language models (LLMs), which is expected to be retraining-free to avoid the enormous retraining cost. This retraining-free paradigm involves (1) pruning criteria to define the architecture and (2) distortion reconstruction to restore performance. However, existing methods often emphasize pruning criteria while using reconstruction techniques that are specific to certain modules or criteria, resulting in limited generalizability. To address this, we introduce the Linear Interpolation-based Adaptive Reconstruction (LIAR) framework, which is both efficient and effective. LIAR does not require back-propagation or retraining and is compatible with various pruning criteria and modules. By applying linear interpolation to the preserved weights, LIAR minimizes reconstruction error and effectively reconstructs the pruned output. Our evaluations on benchmarks such as GLUE, SQuAD, WikiText, and common sense reasoning show that LIAR enables a BERT model to maintain 98% accuracy even after removing 50% of its parameters and achieves top performance for LLaMA in just a few minutes.

cs.LG↗

arXiv2024

OVLW-DETR: Open-Vocabulary Light-Weighted Detection Transformer

Open-vocabulary object detection focusing on detecting novel categories guided by natural language. In this report, we propose Open-Vocabulary Light-Weighted Detection Transformer (OVLW-DETR), a deployment friendly open-vocabulary detector with strong performance and low latency. Building upon OVLW-DETR, we provide an end-to-end training recipe that transferring knowledge from vision-language model (VLM) to object detector with simple alignment. We align detector with the text encoder from VLM by replacing the fixed classification layer weights in detector with the class-name embeddings extracted from the text encoder. Without additional fusing module, OVLW-DETR is flexible and deployment friendly, making it easier to implement and modulate. improving the efficiency of interleaved attention computation. Experimental results demonstrate that the proposed approach is superior over existing real-time open-vocabulary detectors on standard Zero-Shot LVIS benchmark. Source code and pre-trained models are available at [https://github.com/Atten4Vis/LW-DETR].

cs.CV↗

arXiv2024

Impact of LRG1 and LRG2 in DESI 2024 BAO data on dark energy evolution

Recent measurements of baryon acoustic oscillations (BAO) by the Dark Energy Spectroscopic Instrument (DESI) suggest a preference for a dynamic dark energy model over a cosmological constant. This conclusion emerges from the combination of DESI's BAO data with observations of the Cosmic Microwave Background (CMB) and various type Ia supernova (SN Ia) catalogues. The deviation observed in the cosmological constant ($Λ$) reflects a departure from the standard cosmological model. Testing this deviation involves examining the consistency between cosmological parameters derived from early and late-time observations. Specifically, we focus on the matter density parameter $ω_m = Ω_mh^2$ and introduce ${\rm ratio}(ω_m)$ to assess consistency, which is defined as the ratio of $ω_m$ values constrained by high and low-redshift measurements. This ratio serves as a metric for quantifying deviations from the $Λ$CDM model. In this paper, we find that the DESI BAO+CMB yields ${\rm ratio}(ω_m)=1.0171\pm0.0066$. Upon excluding the LRG1 and LRG2 data in DESI BAO, this ratio adjusts to ${\rm ratio}(ω_m)=1.0100\pm0.0082$. This shift, corresponding to a change from $2.6σ$ to $1.2σ$, indicates that the deviation from the $Λ$CDM model is predominantly driven by these two samples from the DESI BAO measurements. To substantiate this conclusion, we utilized two cosmological model-independent methods to reconstruct the cosmic expansion history. Both reconstructions of the Hubble parameter $H(z)$ indicate that the evolving features of dark energy are determined by the combined LRG1 and LRG2 data. Therefore, different methods have reached the same conclusion, namely the importance of accurately measuring the BAO feature in LRG1 and LRG2 data.

astro-ph.CO↗

arXiv2024

DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis

Diffusion models have achieved great success in image generation, with the backbone evolving from U-Net to Vision Transformers. However, the computational cost of Transformers is quadratic to the number of tokens, leading to significant challenges when dealing with high-resolution images. In this work, we propose Diffusion Mamba (DiM), which combines the efficiency of Mamba, a sequence model based on State Space Models (SSM), with the expressive power of diffusion models for efficient high-resolution image synthesis. To address the challenge that Mamba cannot generalize to 2D signals, we make several architecture designs including multi-directional scans, learnable padding tokens at the end of each row and column, and lightweight local feature enhancement. Our DiM architecture achieves inference-time efficiency for high-resolution images. In addition, to further improve training efficiency for high-resolution image generation with DiM, we investigate "weak-to-strong" training strategy that pretrains DiM on low-resolution images ($256\times 256$) and then finetune it on high-resolution images ($512 \times 512$). We further explore training-free upsampling strategies to enable the model to generate higher-resolution images (e.g., $1024\times 1024$ and $1536\times 1536$) without further fine-tuning. Experiments demonstrate the effectiveness and efficiency of our DiM. The code of our work is available here: {\url{https://github.com/tyshiwo1/DiM-DiffusionMamba/}}.

cs.CV↗

arXiv2024

A Back-End Electronics Based on Fiber Communication for Small to Medium-Scale Physics Experiments

Many small and medium-sized physics experiments are being conducted worldwide. These experiments have similar requirements for readout electronics, especially the back-end electronics. Some experiments need a trigger logic unit(TLU) to provide timing and synchronous control signals. This paper introduces a back-end electronics design for small and medium-sized physics experiments; it adopts a daughter-motherboard structure integrated TLU function to provide greater flexibility. Different interfaces and protocols can be flexibly selected according to data bandwidth requirements. It supports 32 optical fiber interfaces based on a field-programmable gate array (FPGA) of normal IOs with 400Mbps of data bandwidth for each channel. At the same time, it supports 16 high-speed communication interfaces based on GTX port with several Gbps data bandwidth of each channel. For the TLU function, this design has 8 HDMI interfaces and one RJ45 interface to provide synchronous triggers and other control signals, and it has six analog LEMOs and four digital LEMOs to accept asynchronous signals from an external source. These design specifications can meet the needs of most small and medium-sized experiments. This set of back-end electronics has been successfully used in experiments such as PandaX-III, VLAST, and moungraphy. Moreover, it has successfully conducted beam tests with a readout of the data of VLAST detectors at CERN.

physics.ins-det↗

arXiv2024

A unified approach to the spectral radius, connectivity and edge-connectivity of graphs

For two integers $r\geq 2$ and $h\geq 0$, the \emph{$h$-extra $r$-component connectivity} $κ^h_r(G)$ of a graph $G$ is defined to be the minimum size of a subset of vertices whose removal disconnects $G$, and there are at least $r$ connected components in $G\!-\!S$ and each component has at least $h+1$ vertices. Denote by $\mathcal{G}_{n,δ}^{κ_r^h}$ the set of graphs with $h$-extra $r$-component connectivity $κ^h_r(G)$ and minimum degree $δ$. The following problem concerning spectral radius was proposed by Brualdi and Solheid [On the spectral radius of complementary acyclic matrices of zeros and one, SIAM J. Algebra Discrete Methods 7 (1986) 265-272]: Given a set of graphs $\mathscr{S}$, find an upper bound for the spectral radius of graphs in $\mathscr{S}$ and characterize the graphs in which the maximal spectral radius is attained. We study this question for $\mathscr{S}=\mathcal{G}_{n,δ}^{κ_r^h}$ where $r\geq 2$ and $h\geq 0$. Fan, Gu and Lin [$l$-connectivity, $l$-edge-connectivity and spectral radius of graphs, \emph{arXiv}:2309.05247] give the answer to $r\geq 2$ and $h=0$. In this paper, we solve this problem completely for $r\geq 2$ and $h\geq1$. Moreover, we also investigate analogous problems for the edge version. Our results can break the restriction of the extremum structure of the conditional connectivity. This implies some previous results in connectivity and edge-connectivity.

math.CO↗

arXiv2024

ISWSST: Index-space-wave State Superposition Transformers for Multispectral Remotely Sensed Imagery Semantic Segmentation

Currently the semantic segmentation task of multispectral remotely sensed imagery (MSRSI) faces the following problems: 1) Usually, only single domain feature (i.e., space domain or frequency domain) is considered; 2) downsampling operation in encoder generally leads to the accuracy loss of edge extraction; 3) multichannel features of MSRSI are not fully considered; and 4) prior knowledge of remote sensing is not fully utilized. To solve the aforementioned issues, an index-space-wave state superposition Transformer (ISWSST) is the first to be proposed for MSRSI semantic segmentation by the inspiration from quantum mechanics, whose superiority is as follows: 1) index, space and wave states are superposed or fused to simulate quantum superposition by adaptively voting decision (i.e., ensemble learning idea) for being a stronger classifier and improving the segmentation accuracy; 2) a lossless wavelet pyramid encoder-decoder module is designed to losslessly reconstruct image and simulate quantum entanglement based on wavelet transform and inverse wavelet transform for avoiding the edge extraction loss; 3) combining multispectral features (i.e. remote sensing index and channel attention mechanism) is proposed to accurately extract ground objects from original resolution images; and 4) quantum mechanics are introduced to interpret the underlying superiority of ISWSST. Experiments show that ISWSST is validated and superior to the state-of-the-art architectures for the MSRSI segmentation task, which improves the segmentation and edge extraction accuracy effectively. Codes will be available publicly after our paper is accepted.

cs.CV↗

arXiv2024

A new subclass of gamma-ray burst originating from compact binary merger

Type I gamma-ray bursts (GRBs) are believed to originate from compact binary merger usually with duration less than 2 seconds for the main emission. However, recent observations of GRB 211211A and GRB 230307A indicate that some merger-origin GRBs could last much longer. Since they show strikingly similar properties (indicating a common mechanism) which are different from the classic "long"-short burst (e.g. GRB 060614), forming an interesting subclass of type I GRBs, we suggest to name them as type IL GRBs. By identifying the first peak of GRB 230307A as a quasi-thermal precursor, we find that the prompt emission of type IL GRB is composed of three episodes: (1) a precursor followed by a short quiescent (or weak emission) period, (2) a long-duration main emission, and (3) an extended emission. With this burst pattern, a good candidate, GRB 170228A, was found in the Fermi/GBM archive data, and subsequent temporal and spectral analyses indeed show that GRB 170228A falls in the same cluster with GRB 211211A and GRB 230307A in many diagnostic figures. Thus this burst pattern could be a good reference for rapidly identifying type IL GRB and conducting low-latency follow-up observation. We estimated the occurrence rate and discussed the physical origins and implications for the three emission episodes of type IL GRBs. Our analysis suggests the pre-merger precursor model, especially the super flare model, is more favored for type IL GRBs.

astro-ph.HE↗

arXiv2024

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs

The rapid advancement of large language models (LLMs) has led to architectures with billions to trillions of parameters, posing significant deployment challenges due to their substantial demands on memory, processing power, and energy consumption. Sparse Mixture-of-Experts (SMoE) architectures have emerged as a solution, activating only a subset of parameters per token, thereby achieving faster inference while maintaining performance. However, SMoE models still face limitations in broader deployment due to their large parameter counts and significant GPU memory requirements. In this work, we introduce a gradient-free evolutionary strategy named EEP (Efficient Expert P}runing) to enhance the pruning of experts in SMoE models. EEP relies solely on model inference (i.e., no gradient computation) and achieves greater sparsity while maintaining or even improving performance on downstream tasks. EEP can be used to reduce both the total number of experts (thus saving GPU memory) and the number of active experts (thus accelerating inference). For example, we demonstrate that pruning up to 75% of experts in Mixtral $8\times7$B-Instruct results in a substantial reduction in parameters with minimal performance loss. Remarkably, we observe improved performance on certain tasks, such as a significant increase in accuracy on the SQuAD dataset (from 53.4% to 75.4%), when pruning half of the experts. With these results, EEP not only lowers the barrier to deploying SMoE models,but also challenges the conventional understanding of model pruning by showing that fewer experts can lead to better task-specific performance without any fine-tuning. Code is available at https://github.com/imagination-research/EEP.

cs.LG↗

arXiv2024

HATs: Hierarchical Adaptive Taxonomy Segmentation for Panoramic Pathology Image Analysis

Panoramic image segmentation in computational pathology presents a remarkable challenge due to the morphologically complex and variably scaled anatomy. For instance, the intricate organization in kidney pathology spans multiple layers, from regions like the cortex and medulla to functional units such as glomeruli, tubules, and vessels, down to various cell types. In this paper, we propose a novel Hierarchical Adaptive Taxonomy Segmentation (HATs) method, which is designed to thoroughly segment panoramic views of kidney structures by leveraging detailed anatomical insights. Our approach entails (1) the innovative HATs technique which translates spatial relationships among 15 distinct object classes into a versatile "plug-and-play" loss function that spans across regions, functional units, and cells, (2) the incorporation of anatomical hierarchies and scale considerations into a unified simple matrix representation for all panoramic entities, (3) the adoption of the latest AI foundation model (EfficientSAM) as a feature extraction tool to boost the model's adaptability, yet eliminating the need for manual prompt generation in conventional segment anything model (SAM). Experimental findings demonstrate that the HATs method offers an efficient and effective strategy for integrating clinical insights and imaging precedents into a unified segmentation model across more than 15 categories. The official implementation is publicly available at https://github.com/hrlblab/HATs.

eess.IV↗

arXiv2024

Essential connectivity and spectral radius of graphs

A graph is trivial if it contains one vertex and no edges. The essential connectivity $κ^{\prime}$ of $G$ is defined to be the minimum number of vertices of $G$ whose removal produces a disconnected graph with at least two non-trivial components. Let $\mathcal{A}_n^{κ',δ}$ be the set of graphs of order $n$ with minimum degree $δ$ and essential connectivity $κ'$. In this paper, we determine the graphs attaining the maximum spectral radii among all graphs in $\mathcal{A}_n^{κ',δ}$ and characterize the corresponding extremal graphs. In addition, we also determine the digraphs which achieve the maximum spectral radii among all strongly connected digraphs with given essential connectivity and give the exact values of the spectral radii of these digraphs.

math.CO↗

arXiv2024

Interband and intraband transitions, as well as charge mobility in driven two-band model with electron phonon coupling

Interband and intraband transitions are fundamental concepts in the study of electronic properties of materials, particularly semiconductors and nanomaterials. These transitions involve the movement of electrons between distinct energy states or bands within a material. Besides, charge mobility is also a critical parameter in materials science and electronics. A thorough understanding of these transitions and mobility is critical for the development and optimization of advanced electronic and optoelectronic devices. In this study, we investigate the influence of external periodic drivings on interband and intraband transitions, as well as charge mobility, within a driven two-band model that includes electron-phonon coupling. These external periodic drivings can include a periodic laser field, a time-varying magnetic or electric field, or an alternating current (AC) voltage source. We have developed the Floquet surface hopping (FSH) and Floquet mean field (FMF) methods to simulate electronic dynamics under various drivings in both real and reciprocal spaces. Our findings demonstrate that periodic drivings can enhance interband transitions while suppressing intraband transitions. Additionally, charge mobility is restrained by these external periodic drivings in the driven two-band model.

cond-mat.mes-hall↗

arXiv2024

Intensity Confusion Matters: An Intensity-Distance Guided Loss for Bronchus Segmentation

Automatic segmentation of the bronchial tree from CT imaging is important, as it provides structural information for disease diagnosis. Despite the merits of previous automatic bronchus segmentation methods, they have paied less attention to the issue we term as \textit{Intensity Confusion}, wherein the intensity values of certain background voxels approach those of the foreground voxels within bronchi. Conversely, the intensity values of some foreground voxels are nearly identical to those of background voxels. This proximity in intensity values introduces significant challenges to neural network methodologies. To address the issue, we introduce a novel Intensity-Distance Guided loss function, which assigns adaptive weights to different image voxels for mining hard samples that cause the intensity confusion. The proposed loss estimates the voxel-level hardness of samples, on the basis of the following intensity and distance priors. We regard a voxel as a hard sample if it is in: (1) the background and has an intensity value close to the bronchus region; (2) the bronchus region and is of higher intensity than most voxels inside the bronchus; (3) the background region and at a short distance from the bronchus. Extensive experiments not only show the superiority of our method compared with the state-of-the-art methods, but also verify that tackling the intensity confusion issue helps to significantly improve bronchus segmentation. Project page: https://github.com/lhaof/ICM.

eess.IV↗