arXiv subjects

Weipeng Chen

Publications and source records attributed to Weipeng Chen.

53 records · Page 3Linked to original sources

arXiv2024

Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding

Large Language Models (LLMs) frequently suffer from inefficiencies, largely attributable to the discord between the requirements of auto-regressive decoding and the architecture of contemporary GPUs. Recently, regressive lightweight speculative decoding has garnered attention for its notable efficiency improvements in text generation tasks. This approach utilizes a lightweight regressive draft model, like a Recurrent Neural Network (RNN) or a single transformer decoder layer, leveraging sequential information to iteratively predict potential tokens. Specifically, RNN draft models are computationally economical but tend to deliver lower accuracy, while attention decoder layer models exhibit the opposite traits. This paper presents Clover-2, an advanced iteration of Clover, an RNN-based draft model designed to achieve comparable accuracy to that of attention decoder layer models while maintaining minimal computational overhead. Clover-2 enhances the model architecture and incorporates knowledge distillation to increase Clover's accuracy and improve overall efficiency. We conducted experiments using the open-source Vicuna 7B and LLaMA3-Instruct 8B models. The results demonstrate that Clover-2 surpasses existing methods across various model architectures, showcasing its efficacy and robustness.

cs.CL↗

arXiv2024

MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic

The advent of large language models (LLMs) like GPT-4 has catalyzed the exploration of multi-task learning (MTL), in which a single model demonstrates proficiency across diverse tasks. Task arithmetic has emerged as a cost-effective approach for MTL. It enables performance enhancement across multiple tasks by adding their corresponding task vectors to a pre-trained model. However, the current lack of a method that can simultaneously achieve optimal performance, computational efficiency, and data privacy limits their application to LLMs. In this paper, we propose \textbf{M}odel \textbf{E}xclusive \textbf{T}ask \textbf{A}rithmetic for merging \textbf{GPT}-scale models, which formalizes the objective of model merging into a multi-task learning framework, aiming to minimize the average loss difference between the merged model and each individual task model. Since data privacy limits the use of multi-task training data, we leverage LLMs' local linearity and task vectors' orthogonality to separate the data term and scaling coefficients term and derive a model-exclusive task arithmetic method. Our proposed MetaGPT is data-agnostic and bypasses the heavy search process, making it cost-effective and easy to implement for LLMs.Extensive experiments demonstrate that MetaGPT leads to improvements in task arithmetic and achieves state-of-the-art performance on multiple tasks.

cs.CL↗

arXiv2024

Towards Event-oriented Long Video Understanding

With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at https://github.com/RUCAIBox/Event-Bench.

cs.CV↗

arXiv2024

Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models

Synthetic data has been proposed as a solution to address the issue of high-quality data scarcity in the training of large language models (LLMs). Studies have shown that synthetic data can effectively improve the performance of LLMs on downstream benchmarks. However, despite its potential benefits, our analysis suggests that there may be inherent flaws in synthetic data. The uniform format of synthetic data can lead to pattern overfitting and cause significant shifts in the output distribution, thereby reducing the model's instruction-following capabilities. Our work delves into these specific flaws associated with question-answer (Q-A) pairs, a prevalent type of synthetic data, and presents a method based on unlearning techniques to mitigate these flaws. The empirical results demonstrate the effectiveness of our approach, which can reverse the instruction-following issues caused by pattern overfitting without compromising performance on benchmarks at relatively low cost. Our work has yielded key insights into the effective use of synthetic data, aiming to promote more robust and efficient LLM training.

cs.CL↗

arXiv2024

Full-ECE: A Metric For Token-level Calibration on Large Language Models

Deep Neural Networks (DNNs) excel in various domains but face challenges in providing accurate uncertainty estimates, which are crucial for high-stakes applications. Large Language Models (LLMs) have recently emerged as powerful tools, demonstrating exceptional performance in language tasks. However, traditional calibration metrics such as Expected Calibration Error (ECE) and classwise-ECE (cw-ECE) are inadequate for LLMs due to their vast vocabularies, data complexity, and distributional focus. To address this, we propose a novel calibration concept called full calibration and introduce its corresponding metric, Full-ECE. Full-ECE evaluates the entire predicted probability distribution, offering a more accurate and robust measure of calibration for LLMs.

cs.CL↗

arXiv2024

Base of RoPE Bounds Context Length

Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the \textit{base} parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.

cs.CL↗

arXiv2024

Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge

Large language models (LLMs) suffer from low efficiency as the mismatch between the requirement of auto-regressive decoding and the design of most contemporary GPUs. Specifically, billions to trillions of parameters must be loaded to the GPU cache through its limited memory bandwidth for computation, but only a small batch of tokens is actually computed. Consequently, the GPU spends most of its time on memory transfer instead of computation. Recently, parallel decoding, a type of speculative decoding algorithms, is becoming more popular and has demonstrated impressive efficiency improvement in generation. It introduces extra decoding heads to large models, enabling them to predict multiple subsequent tokens simultaneously and verify these candidate continuations in a single decoding step. However, this approach deviates from the training objective of next token prediction used during pre-training, resulting in a low hit rate for candidate tokens. In this paper, we propose a new speculative decoding algorithm, Clover, which integrates sequential knowledge into the parallel decoding process. This enhancement improves the hit rate of speculators and thus boosts the overall efficiency. Clover transmits the sequential knowledge from pre-speculated tokens via the Regressive Connection, then employs an Attention Decoder to integrate these speculated tokens. Additionally, Clover incorporates an Augmenting Block that modifies the hidden states to better align with the purpose of speculative generation rather than next token prediction. The experiment results demonstrate that Clover outperforms the baseline by up to 91% on Baichuan-Small and 146% on Baichuan-Large, respectively, and exceeds the performance of the previously top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on Baichuan-Large, respectively.

cs.CL↗

arXiv2024

Examining the possibility of chiral superconductivity in Sr$_2$RuO$_4$ and other compounds via applied supercurrent

One approach to probe the still controversial superconductivity in Sr$_2$RuO$_4$ is to apply external perturbations that break the underlying tetragonal crystalline symmetry. Chiral $p_x+ip_y$ and $d_{xz}+id_{yz}$ states respond to such perturbations in ways that may help to distinguish them from other superconducting pairings. However, past experimental efforts along this line, using uniaxial strains and magnetic fields parallel to the RuO$_2$ plane, have not been able to reach an unambiguous conclusion. In this study, we propose to further examine the possibility of chiral superconducting order in Sr$_2$RuO$_4$ using an alternative tetragonal-symmetry-breaking perturbation -- in-plane supercurrent. We study the superconducting phase diagram as a function of both temperature and the applied supercurrent. Supercurrent generically splits the transition of the two chiral order parameter components, and we show that the splitting can give rise to visible specific heat anomalies. Furthermore, supercurrent parallel and anti-parallel to the unidirectional propagation of the chiral edge modes impact the edge states in different manner. This difference manifests in the tunneling spectrum, thereby providing an additional means to probe the chirality even when the related spontaneous edge current is vanishingly small. Finally, we discuss the distinction of supercurrent responses in non-chiral time-reversal-symmetry-breaking superconducting states. Our proposal can be applied to other candidate chiral superconductors.

cond-mat.supr-con↗

arXiv2024

Quantum-geometry-induced anomalous Hall effect in nonunitary superconductors and application to Sr$_2$RuO$_4$

The polar Kerr effect and the closely related anomalous charge Hall effect are among the most distinguishing signatures of the superconducting state in Sr$_2$RuO$_4$, as well as in several other compounds. These effects are often thought to be derived from chiral superconducting pairing, and different mechanisms have been invoked for the explanation. However, the intrinsic mechanisms proposed previously often involve unrealistically strong interband Cooper pairing. In this study we show that, even without interband pairing, nonunitary superconducting states can support intrinsic anomalous charge Hall effect, thanks to the quantum geometric properties of the Bloch electrons. The key here is to have a normal-state spin Hall effect, for which a nonzero spin-orbit coupling is essential. A finite charge Hall effect then naturally arises at the onset of a spin-polarized nonunitary superconducting pairing. It depends on both the spin polarization and the normal-state electron Berry curvature, the latter of which is the imaginary part of the quantum geometric tensor of the Bloch states. Applying our results to the weakly-paired Sr$_2$RuO$_4$ we conclude that, if the reported Kerr effect is of intrinsic origin, the superconducting state is most likely nonunitary and has odd-parity. Our theory may be generalized to other superconductors that exhibit polar Kerr effect.

cond-mat.supr-con↗

arXiv2023

Superconducting Diode Effect and Large Magnetochiral Anisotropy in T$_d$-MoTe$_2$ Thin Film

In the absence of time-reversal invariance, metals without inversion symmetry may exhibit nonreciprocal charge transport -- a magnetochiral anisotropy that manifests as unequal electrical resistance for opposite current flow directions. If superconductivity also sets in, the charge transmission may become dissipationless in one direction while remaining dissipative in the opposite, thereby realizing a superconducting diode. Through both direct-current and alternating-current measurements, we study the nonreciprocal effects in thin films of the noncentrosymmetric superconductor T$_d$-MoTe\textsubscript{2} with disorders. We observe nonreciprocal superconducting critical currents with a diode efficiency close to 20\%~, and a large magnetochiral anisotropy coefficient up to $\SI{5.9e8}{\per\tesla\per\ampere}$, under weak out-of-plane magnetic field in the millitesla range. The great enhancement of rectification efficiency under out-of-plane magnetic field is likely abscribed to the vortex ratchet effect, which naturally appears in the noncentrosymmetric superconductor with disorders. Intriguingly, unlike the finding in Rashba systems, the strongest in-plane nonreciprocal effect does not occur when the field is perpendicular to the current flow direction. We develop a phenomenological theory to demonstrate that this peculiar behavior can be attributed to the asymmetric structure of spin-orbit coupling in T$_d$-MoTe\textsubscript{2}. Our study highlights how the crystallographic symmetry critically impacts the nonreciprocal transport, and would further advance the research for designing the superconducting diode with the best performance.

cond-mat.supr-con↗

arXiv2023

Pair density wave facilitated by Bloch quantum geometry in nearly flat band multiorbital superconductors

Bloch electrons in multiorbital systems carry quantum geometric information characteristic of their wavevector-dependent interorbital mixing. The geometric nature impacts electromagnetic responses, and this effect carries over to the superconducting state, which receives a geometric contribution to the superfluid weight. In this paper, we show that this contribution could become negative under certain appropriate circumstances. This may facilitate the stabilization of Cooper pairings with real space phase modulation, i.e. the pair density wave order, as we demonstrate through two-orbital model Bogoliubov de-Gennes mean-field calculations. The quantum geometric effect therefore constitutes an intrinsic mechanism for the formation of such a novel phase of matter in the absence of external magnetic field.

cond-mat.supr-con↗

arXiv2023

Impact of random impurities on the anomalous Hall effect in chiral superconductors

The anomalous Hall effect and the closely related polar Kerr effect are among the most direct evidence of chiral Cooper pairing in some superconductors. While it has been known that disorder or multiband pairing is typically needed for these effects to manifest, there is a lack of direct real-space investigation with regard to how disorder impacts the Hall response in both single-band and multiband chiral superconductors. On the basis of chiral superconducting models often adopted for \SRO, we study in this work the anomalous Hall effect in the presence of random non-magnetic impurities on real-space lattices. The single-band chiral p-wave ($p_x+ip_y$) calculation qualitatively reproduces the Hall conductivity obtained in previous skew-scattering-type diagrammatic analyses, along with some quantitative difference originating primarily from contributions involving impurity-induced in-gap states. The non-p-wave chiral states, such as $d_{x^2-y^2}+id_{xy}$, generically exhibit finite Hall response in the presence of random impurities, in contrast to a conclusion drawn from the aforementioned diagrammatic study. In particular, while pointlike impurities appears to induce minuscule Hall conductivity in non-self-consistent calculations, self-consistency and finite-range impurity potentials can both lead to substantial Hall conductivity. On the other hand, the intrinsic Hall conductivity in multiband chiral superconductors, which is related to interband transitions, decreases parametrically as disorder suppresses the superconducting order parameter. In addition, we check that random impurities do not induce anomalous Hall effect in non-chiral but time-reversal symmetry breaking superconducting states the likes of $s+i d_{x^2-y^2}$ and $d_{x^2-y^2}+i g_{xy(x^2-y^2)}$. We briefly remark on the implications of our results for Kerr effect measurements.

cond-mat.supr-con↗

arXiv2021

Quantum-geometry-induced intrinsic optical anomaly in multiorbital superconductors

Bloch electrons in multiorbital systems carry nontrivial quantum geometric information characteristic of their orbital composition as a function of their wave vector. When such electrons form Cooper pairs, the resultant superconducting state naturally inherits aspects of the quantum geometry. In this paper, we study how this geometric character is revealed in the intrinsic optical response of the superconducting state. In particular, due to the superconducting gap opening, interband optical transitions involving states around the Fermi level are forbidden. This generally leads to an anomalous suppression of the optical conductivity at frequencies matching the band separation -- which could be significantly higher than the superconducting gap energy. We discuss how the predicted anomaly may have already emerged in two earlier measurements on an iron-based superconductor. When interband Cooper pairing is present, intraband optical transitions may be allowed and finite conductivity emerges at low frequencies right above the gap edge. These conductivities depend crucially on the off-diagonal elements of the velocity matrix in the band representation, i.e. the interband velocity -- which is related to the non-Abelian Berry connection of the Bloch states.

cond-mat.supr-con↗

arXiv2021

ComQA:Compositional Question Answering via Hierarchical Graph Neural Networks

With the development of deep learning techniques and large scale datasets, the question answering (QA) systems have been quickly improved, providing more accurate and satisfying answers. However, current QA systems either focus on the sentence-level answer, i.e., answer selection, or phrase-level answer, i.e., machine reading comprehension. How to produce compositional answers has not been throughout investigated. In compositional question answering, the systems should assemble several supporting evidence from the document to generate the final answer, which is more difficult than sentence-level or phrase-level QA. In this paper, we present a large-scale compositional question answering dataset containing more than 120k human-labeled questions. The answer in this dataset is composed of discontiguous sentences in the corresponding document. To tackle the ComQA problem, we proposed a hierarchical graph neural networks, which represents the document from the low-level word to the high-level sentence. We also devise a question selection and node selection task for pre-training. Our proposed model achieves a significant improvement over previous machine reading comprehension methods and pre-training methods. Codes and dataset can be found at \url{https://github.com/benywon/ComQA}.

cs.CL↗

arXiv2020

Inter-orbital p- and d-wave pairings between $d_{xz/yz}$ and $d_{xy}$ orbitals in Sr$_2$RuO$_4$

We study the pairing symmetry of Sr$_2$RuO$_4$ through the group-theoretical approach. We emphasize the role of pairing interaction between the quasi-one-dimensional(Q1D) $d_{xz/yz}$ and quasi-two-dimensional(Q2D) $d_{xy}$ orbitals. It is found that two degenerate inter-orbital time-reversal-invariant(TRI) p-wave pairings, one is spin-singlet and the other spin-triplet with out-of-plane $\bm{d}$-vector, could be the most promising candidates. Several important physical quantities are presented, including the near-nodal gap structure, the unchanged out-of-plane Knight shift, and no split transition under strain, which are consistent with the experiments. In addition, these p-wave pairings shed light on resolving the contradiction between the time-reversal breaking and reduced in-plane Knight shift measurements. As the system reaches the Van Hove singularity under applied strain, the pairing symmetry would become a d-wave pairing mainly consisting of inter-orbital components, which could be responsible for the strained 3$K$ phase.

cond-mat.supr-con↗

arXiv2019

Spin and charge transport in topological nodal-line semimetals

We study transport properties of topological Weyl nodal-line semimetals(NLSs). Starting from a minimal lattice model with a single nodal loop, and by focusing on a normal-metal-NLS-normal-metal junction, we investigate the dependence of the novel transport behavior on the orientation of the nodal loop. When the loop is parallel to the junction interfaces, the transmitted current is found to be nearly fully spin-polarized. Correspondingly, there exists a spin orientation, along which the incident electrons would be totally reflected. An unusual resonance of half transmission with the participation of surface states also occurs for a pair of incident electrons with opposite spin orientations. All these phenomena have been shown to originate from the existence of a single forward-propagating mode in the NLS of the junction, and argued to survive in more generic multi-band Weyl NLSs.

cond-mat.mes-hall↗

arXiv2019

Topological Ising pairing states in monolayer and trilayer TaS$_2$

We study the possibility of topological superconductivity in the noncentrosymmetric monolayer and trilayer TaS$_2$ with out-of-plane mirror symmetry. A gapless time-reversal invariant f+s-wave pairing state with even mirror parity is found to be a promising candidate. This mixing state holds 12(36) nodes at the Fermi pockets around $Γ$ for monolayer(trilayer) case and its unconventional superconductivity is consistent with the STM experiments observed in 2$H$-TaS$_2$ thin flakes. Furthermore, with doping or under uniaxial pressure for trilayer 2$H$-TaS$_2$, large-Chern-number time-reversal symmetry breaking mixing states between d+id- and p-ip-wave pairings can be realized in the phase diagram.

cond-mat.supr-con↗