arXiv subjects

Yu Wang

Publications and source records attributed to Yu Wang.

At least 685 records · Page 38Linked to original sources

arXiv2024

Unbiased approximation of the ergodic measure for piecewise $α$-stable Ornstein-Uhlenbeck processes arising in queueing networks

Piecewise $α$-stable Ornstein-Uhlenbeck (OU) processes arising in queue networks usually do not have an explicit dissipation, which makes the related numerical methods such as Euler-Maruyama (EM) scheme more difficult to analyze. We develop an EM scheme with decreasing step size $Λ=(η_n)_{n\in \mathbb{N}}$ to approximate their ergodic measures. This approximation does not have a bias and has a rate $η^{1/α}_n$ in Wasserstein-1 distance. We show by the classical OU process that our convergence rate is optimal. We further prove the central limit theorem (CLT) and moderate derivation principle (MDP) for the empirical measure of these piecewise $α$-stable Ornstein-Uhlenbeck processes. In addition, we use the Sinkhorn--Knopp algorithm to compute the Wasserstein-1 distance and conduct simulations for several concrete examples.

math.PR↗

arXiv2024

GRS-QA -- Graph Reasoning-Structured Question Answering Dataset

Large Language Models (LLMs) have excelled in multi-hop question-answering (M-QA) due to their advanced reasoning abilities. However, the impact of the inherent reasoning structures on LLM M-QA performance remains unclear, largely due to the absence of QA datasets that provide fine-grained reasoning structures. To address this gap, we introduce the Graph Reasoning-Structured Question Answering Dataset (GRS-QA), which includes both semantic contexts and reasoning structures for QA pairs. Unlike existing M-QA datasets, where different reasoning structures are entangled together, GRS-QA explicitly captures intricate reasoning pathways by constructing reasoning graphs, where nodes represent textual contexts and edges denote logical flows. These reasoning graphs of different structures enable a fine-grained evaluation of LLM reasoning capabilities across various reasoning structures. Our empirical analysis reveals that LLMs perform differently when handling questions with varying reasoning structures. This finding facilitates the exploration of textual structures as compared with semantics.

cs.CL↗

arXiv2024

Inverse problems for stochastic partial differential equations

This book aims to provide a brief overview of recent advancements in the theory of inverse problems for stochastic partial differential equations. In order to keep the content concise, we will only discuss the inverse problems of two typical classes of stochastic partial differential equations: second-order stochastic parabolic equations and secondorder stochastic hyperbolic equations. The main tool for studying these inverse problem is Carleman estimate. We do not intend to pursue any general treatment of the Carleman estimates themselves and choose direct arguments based on basic stochastic calculus, rather than more general sophisticated methods. As this field is still developing and there are many challenging issues to be addressed, the purpose of this book is not to serve as a comprehensive summary, but rather to spark interest and encourage further exploration in this area among readers. We prefer to present results that, from our perspective, include fresh and promising ideas. In cases where a complete mathematical theory is lacking, we only provide the available results. We do not intend for the current book to be encyclopedic in any sense, and the references are limited.

math.PR↗

arXiv2024

Selecting Between BERT and GPT for Text Classification in Political Science Research

Political scientists often grapple with data scarcity in text classification. Recently, fine-tuned BERT models and their variants have gained traction as effective solutions to address this issue. In this study, we investigate the potential of GPT-based models combined with prompt engineering as a viable alternative. We conduct a series of experiments across various classification tasks, differing in the number of classes and complexity, to evaluate the effectiveness of BERT-based versus GPT-based models in low-data scenarios. Our findings indicate that while zero-shot and few-shot learning with GPT models provide reasonable performance and are well-suited for early-stage research exploration, they generally fall short - or, at best, match - the performance of BERT fine-tuning, particularly as the training set reaches a substantial size (e.g., 1,000 samples). We conclude by comparing these approaches in terms of performance, ease of use, and cost, providing practical guidance for researchers facing data limitations. Our results are particularly relevant for those engaged in quantitative text analysis in low-resource settings or with limited labeled data.

cs.CL↗

arXiv2024

Robust estimation for high-dimensional time series with heavy tails

We study in this paper the problem of least absolute deviation (LAD) regression for high-dimensional heavy-tailed time series which have finite $α$-th moment with $α\in (1,2]$. To handle the heavy-tailed dependent data, we propose a Catoni type truncated minimization problem framework and obtain an $\mathcal{O}\big( \big( (d_1+d_2) (d_1\land d_2) \log^2 n / n \big)^{(α- 1)/α} \big)$ order excess risk, where $d_1$ and $d_2$ are the dimensionality and $n$ is the number of samples. We apply our result to study the LAD regression on high-dimensional heavy-tailed vector autoregressive (VAR) process. Simulations for the VAR($p$) model show that our new estimator with truncation are essential because the risk of the classical LAD has a tendency to blow up. We further apply our estimation to the real data and find that ours fits the data better than the classical LAD.

math.ST↗

arXiv2024

Extremal spectral radius and $g$-good $r$-component connectivity

For $F\subseteq V(G)$, if $G-F$ is a disconnected graph with at least $r$ components and each vertex $v\in V(G)\backslash F$ has at least $g$ neighbors, then $F$ is called a $g$-good $r$-component cut of $G$. The $g$-good $r$-component connectivity of $G$, denoted by $cκ_{g,r}(G)$, is the minimum cardinality of $g$-good $r$-component cuts of $G$. Let $\mathcal{G}_n^{k,δ}$ be the set of graphs of order $n$ with minimum degree $δ$ and $g$-good $r$-component connectivity $cκ_{g,r}(G)=k$. In the paper, we determine the extremal graphs attaining the maximum spectral radii among all graphs in $\mathcal{G}_n^{k,δ}$. A subset $F\subseteq V(G)$ is called a $g$-good neighbor cut of $G$ if $G-F$ is disconnected and each vertex $v\in V(G)\backslash F$ has at least $g$ neighbors. The $g$-good neighbor connectivity $κ_g(G)$ of a graph $G$ is the minimum cardinality of $g$-good neighbor cuts of $G$. The condition of $g$-good neighbor connectivity is weaker than that of $g$-good $r$-component connectivity, and there is no requirement on the number of components. As a counterpart, we also study similar problem for $g$-good neighbor connectivity.

math.CO↗

arXiv2024

Efficient and Effective Retrieval of Dense-Sparse Hybrid Vectors using Graph-based Approximate Nearest Neighbor Search

ANNS for embedded vector representations of texts is commonly used in information retrieval, with two important information representations being sparse and dense vectors. While it has been shown that combining these representations improves accuracy, the current method of conducting sparse and dense vector searches separately suffers from low scalability and high system complexity. Alternatively, building a unified index faces challenges with accuracy and efficiency. To address these issues, we propose a graph-based ANNS algorithm for dense-sparse hybrid vectors. Firstly, we propose a distribution alignment method to improve accuracy, which pre-samples dense and sparse vectors to analyze their distance distribution statistic, resulting in a 1%$\sim$9% increase in accuracy. Secondly, to improve efficiency, we design an adaptive two-stage computation strategy that initially computes dense distances only and later computes hybrid distances. Further, we prune the sparse vectors to speed up the calculation. Compared to naive implementation, we achieve $\sim2.1\times$ acceleration. Thorough experiments show that our algorithm achieves 8.9x$\sim$11.7x throughput at equal accuracy compared to existing hybrid vector search algorithms.

cs.IR↗

arXiv2024

A Compact Readout Electronics Based on Current Amplifier for Micromegas Detector in Muon Imaging

Muon imaging technology is an innovative imaging technique that can be applied in volcano imaging, heavy nuclear material detection, and archaeological research. The Micromegas detector is a promising choice for muon imaging due to its high spatial resolution and large area. However, the large number of readout channels poses a challenge for electronics. In this paper, a compact front-end electronics (FEE) for reading Micromegas detectors is presented. The electronics use the commercial current-to-digital readout chip, ADAS1128, which integrates 128 current amplifiers for multi-channel charge measurement. After verifying the basic performance of the electronics, the energy resolution was obtained with a radioactive source. Furthermore, a muon imaging system prototype was set up and its spatial resolution was evaluated in a test with cosmic ray muons. The system prototype can reconstruct the boundaries of sufficiently massive objects with a size of 2 cm in a scattering imaging test.

physics.ins-det↗

arXiv2024

A Survey of Small Language Models

Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources, making them ideal for various settings including on-device, mobile, edge devices, among many others. In this article, we present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques. We propose a novel taxonomy for categorizing the methods used to optimize SLMs, including model compression, pruning, and quantization techniques. We summarize the benchmark datasets that are useful for benchmarking SLMs along with the evaluation metrics commonly used. Additionally, we highlight key open challenges that remain to be addressed. Our survey aims to serve as a valuable resource for researchers and practitioners interested in developing and deploying small yet efficient language models.

cs.CL↗

arXiv2024

Beyond the Phase Ordering Problem: Finding the Globally Optimal Code w.r.t. Optimization Phases

In this paper, we propose a new concept called \textit{semantically equivalence} \wrt \textit{optimization phases} \textit{(\sep)}, which defines the set of programs a compiler considers semantically equivalent to the input using a set of optimization phases. We show both theoretically and empirically that solving the phase ordering problem does not necessarily result in the most efficient code among all programs that a compiler deems semantically equivalent to the input, hereinafter referred to as the global optimal code \wrt optimization phases. To find the global optimal code \wrt optimization phases, we present a conceptual framework, leveraging the reverse of existing optimization phases. In theory, we prove that the framework is capable of finding the global optimal code for any program. We realize this framework into a technique, called \textit{iterative bi-directional optimization (\tool)}, which performs both the normal and reverse optimizations to increase and decrease the efficiency of the generated code, respectively. We evaluate \tool on C/C++ files randomly extracted from highly mature and influential programs (\eg, Linux kernel, OpenSSL, Z3). Results show that \tool frequently generates more efficient code -- measured by either code size or runtime performance -- than exhaustive search, which is the solution to the phase ordering problem. We also find by simply incorporating \tool's reverse optimization phases, the effectiveness of the optimization of state-of-the-art compilers (\eg, GCC/LLVM) can be significantly improved.

cs.PL↗

arXiv2024

DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization

Diffusion models have achieved remarkable progress in the field of image generation due to their outstanding capabilities. However, these models require substantial computing resources because of the multi-step denoising process during inference. While traditional pruning methods have been employed to optimize these models, the retraining process necessitates large-scale training datasets and extensive computational costs to maintain generalization ability, making it neither convenient nor efficient. Recent studies attempt to utilize the similarity of features across adjacent denoising stages to reduce computational costs through simple and static strategies. However, these strategies cannot fully harness the potential of the similar feature patterns across adjacent timesteps. In this work, we propose a novel pruning method that derives an efficient diffusion model via a more intelligent and differentiable pruner. At the core of our approach is casting the model pruning process into a SubNet search process. Specifically, we first introduce a SuperNet based on standard diffusion via adding some backup connections built upon the similar features. We then construct a plugin pruner network and design optimization losses to identify redundant computation. Finally, our method can identify an optimal SubNet through few-step gradient optimization and a simple post-processing procedure. We conduct extensive experiments on various diffusion models including Stable Diffusion series and DiTs. Our DiP-GO approach achieves 4.4 x speedup for SD-1.5 without any loss of accuracy, significantly outperforming the previous state-of-the-art methods.

cs.CV↗

arXiv2024

Error estimates between SGD with momentum and underdamped Langevin diffusion

Stochastic gradient descent with momentum is a popular variant of stochastic gradient descent, which has recently been reported to have a close relationship with the underdamped Langevin diffusion. In this paper, we establish a quantitative error estimate between them in the 1-Wasserstein and total variation distances.

stat.ML↗

arXiv2024

Towards Trustworthy Knowledge Graph Reasoning: An Uncertainty Aware Perspective

Recently, Knowledge Graphs (KGs) have been successfully coupled with Large Language Models (LLMs) to mitigate their hallucinations and enhance their reasoning capability, such as in KG-based retrieval-augmented frameworks. However, current KG-LLM frameworks lack rigorous uncertainty estimation, limiting their reliable deployment in high-stakes applications. Directly incorporating uncertainty quantification into KG-LLM frameworks presents challenges due to their complex architectures and the intricate interactions between the knowledge graph and language model components. To address this gap, we propose a new trustworthy KG-LLM framework, Uncertainty Aware Knowledge-Graph Reasoning (UAG), which incorporates uncertainty quantification into the KG-LLM framework. We design an uncertainty-aware multi-step reasoning framework that leverages conformal prediction to provide a theoretical guarantee on the prediction set. To manage the error rate of the multi-step process, we additionally introduce an error rate control module to adjust the error rate within the individual components. Extensive experiments show that our proposed UAG can achieve any pre-defined coverage rate while reducing the prediction set/interval size by 40% on average over the baselines.

cs.AI↗

arXiv2024

DiTFastAttn: Attention Compression for Diffusion Transformer Models

Diffusion Transformers (DiT) excel at image and video generation but face computational challenges due to the quadratic complexity of self-attention operators. We propose DiTFastAttn, a post-training compression method to alleviate the computational bottleneck of DiT. We identify three key redundancies in the attention computation during DiT inference: (1) spatial redundancy, where many attention heads focus on local information; (2) temporal redundancy, with high similarity between the attention outputs of neighboring steps; (3) conditional redundancy, where conditional and unconditional inferences exhibit significant similarity. We propose three techniques to reduce these redundancies: (1) Window Attention with Residual Sharing to reduce spatial redundancy; (2) Attention Sharing across Timesteps to exploit the similarity between steps; (3) Attention Sharing across CFG to skip redundant computations during conditional generation. We apply DiTFastAttn to DiT, PixArt-Sigma for image generation tasks, and OpenSora for video generation tasks. Our results show that for image generation, our method reduces up to 76% of the attention FLOPs and achieves up to 1.8x end-to-end speedup at high-resolution (2k x 2k) generation.

cs.CV↗

arXiv2024

CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios

Large Language Models (LLMs) have been widely adopted to process long-context tasks. However, the large memory overhead of the key-value (KV) cache poses significant challenges in long-context scenarios. Existing training-free KV cache compression methods typically focus on quantization and token pruning, which have compression limits, and excessive sparsity can lead to severe performance degradation. Other methods design new architectures with less KV overhead but require significant training overhead. To address the above two drawbacks, we further explore the redundancy in the channel dimension and apply an architecture-level design with minor training costs. Therefore, we introduce CSKV, a training-efficient Channel Shrinking technique for KV cache compression: (1) We first analyze the singular value distribution of the KV cache, revealing significant redundancy and compression potential along the channel dimension. Based on this observation, we propose using low-rank decomposition for key and value layers and storing the low-dimension features. (2) To preserve model performance, we introduce a bi-branch KV cache, including a window-based full-precision KV cache and a low-precision compressed KV cache. (3) To reduce the training costs, we minimize the layer-wise reconstruction loss for the compressed KV cache instead of retraining the entire LLMs. Extensive experiments show that CSKV can reduce the memory overhead of the KV cache by 80% while maintaining the model's long-context capability. Moreover, we show that our method can be seamlessly combined with quantization to further reduce the memory overhead, achieving a compression ratio of up to 95%. Code is available at https://github.com/wln20/CSKV.

cs.LG↗

arXiv2024

TAIA: Large Language Models are Out-of-Distribution Data Learners

Fine-tuning on task-specific question-answer pairs is a predominant method for enhancing the performance of instruction-tuned large language models (LLMs) on downstream tasks. However, in certain specialized domains, such as healthcare or harmless content generation, it is nearly impossible to obtain a large volume of high-quality data that matches the downstream distribution. To improve the performance of LLMs in data-scarce domains with domain-mismatched data, we re-evaluated the Transformer architecture and discovered that not all parameter updates during fine-tuning contribute positively to downstream performance. Our analysis reveals that within the self-attention and feed-forward networks, only the fine-tuned attention parameters are particularly beneficial when the training set's distribution does not fully align with the test set. Based on this insight, we propose an effective inference-time intervention method: Training All parameters but Inferring with only Attention (\trainallInfAttn). We empirically validate \trainallInfAttn using two general instruction-tuning datasets and evaluate it on seven downstream tasks involving math, reasoning, and knowledge understanding across LLMs of different parameter sizes and fine-tuning techniques. Our comprehensive experiments demonstrate that \trainallInfAttn achieves superior improvements compared to both the fully fine-tuned model and the base model in most scenarios, with significant performance gains. The high tolerance of \trainallInfAttn to data mismatches makes it resistant to jailbreaking tuning and enhances specialized tasks using general data. Code is available in \url{https://github.com/pixas/TAIA_LLM}.

cs.CL↗

arXiv2024

MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation

Large language models (LLMs) have shown substantial progress in natural language understanding and generation, proving valuable especially in the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks, which can be categorized as knowledge-intensive tasks and alignment-required tasks. Previous approaches either ignore the latter task or focus on a minority of tasks and hence lose generalization. To address these drawbacks, we propose a progressive fine-tuning pipeline. This pipeline employs a Knowledge Aggregator and a Noise aggregator to encode diverse knowledge in the first stage and filter out detrimental information. In the second stage, we drop the Noise Aggregator to avoid the interference of suboptimal representation and leverage an additional alignment module optimized towards an orthogonal direction to the knowledge space to mitigate knowledge forgetting. Based on this two-stage paradigm, we proposed a Medical LLM through decoupling Clinical Alignment and Knowledge Aggregation (MedCare), which is designed to achieve state-of-the-art (SOTA) performance on over 20 medical tasks, as well as SOTA results on specific medical alignment tasks. Various model sizes of MedCare (1.8B, 7B, 14B) all demonstrate significant improvements over existing models with similar model sizes.

cs.CL↗

arXiv2024

SPF-EMPC Planner: A real-time multi-robot trajectory planner for complex environments with uncertainties

In practical applications, the unpredictable movement of obstacles and the imprecise state observation of robots introduce significant uncertainties for the swarm of robots, especially in cluster environments. However, existing methods are difficult to realize safe navigation, considering uncertainties, complex environmental structures, and robot swarms. This paper introduces an extended state model predictive control planner with a safe probability field to address the multi-robot navigation problem in complex, dynamic, and uncertain environments. Initially, the safe probability field offers an innovative approach to model the uncertainty of external dynamic obstacles, combining it with an unconstrained optimization method to generate safe trajectories for multi-robot online. Subsequently, the extended state model predictive controller can accurately track these generated trajectories while considering the robots' inherent model constraints and state uncertainty, thus ensuring the practical feasibility of the planned trajectories. Simulation experiments show a success rate four times higher than that of state-of-the-art algorithms. Physical experiments demonstrate the method's ability to operate in real-time, enabling safe navigation for multi-robot in uncertain environments.

cs.RO↗