arXiv subjects

Yuhao Zhou

Publications and source records attributed to Yuhao Zhou.

At least 91 records · Page 5Linked to original sources

arXiv2025

EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection

Rotary Position Embedding (RoPE) enables each attention head to capture multi-frequency information along the sequence dimension and is widely applied in foundation models. However, the nonlinearity introduced by RoPE complicates optimization of the key state in the Key-Value (KV) cache for RoPE-based attention. Existing KV cache compression methods typically store key state before rotation and apply the transformation during decoding, introducing additional computational overhead. This paper introduces EliteKV, a flexible modification framework for RoPE-based models supporting variable KV cache compression ratios. EliteKV first identifies the intrinsic frequency preference of each head using RoPElite, selectively restoring linearity to certain dimensions of key within attention computation. Building on this, joint low-rank compression of key and value enables partial cache sharing. Experimental results show that with minimal uptraining on only $0.6\%$ of the original training data, RoPE-based models achieve a $75\%$ reduction in KV cache size while preserving performance within a negligible margin. Furthermore, EliteKV consistently performs well across models of different scales within the same family.

cs.LG↗

arXiv2025

SIP-IFVM: A time-evolving coronal model with an extended magnetic field decomposition strategy

Time-evolving magnetohydrodynamic (MHD) coronal modeling, driven by a series of time-dependent photospheric magnetograms, represents a new generation of coronal simulations. This approach offers greater realism compared to traditional coronal models constrained by a static magnetogram. However, its practical application is seriously limited by low computational efficiency and poor numerical stability. Therefore, we propose an extended magnetic field decomposition strategy and implement it in the implicit MHD model to develop a coronal model that is both efficient and numerically stable enough for simulating the long-term evolutions of the global corona. The traditional decomposition strategies split the magnetic field into a time-invariant potential field and a time-dependent component $\mathbf{B}_1$. It works well for quasi-steady-state coronal simulations where $\left|\mathbf{B}_1\right|$ is typically small. However, as the inner-boundary magnetic field evolves, $\left|\mathbf{B}_1\right|$ can grow significantly larger and its discretization errors often lead to nonphysical negative thermal pressure, ultimately causing the code to crash. In this paper, we mitigate such undesired situations by introducing a temporally piecewise-constant variable to accommodate part of the non-potential field and remain $\left|\mathbf{B}_1\right|$ consistently small throughout the simulations. We incorporate this novel magnetic field decomposition strategy into our implicit MHD coronal model and apply it to simulate the evolution of coronal structures within 0.1 AU over two solar-maximum Carrington rotations. The results show that this coronal model effectively captures observations and performs more than 80 times faster than real time using only 192 CPU cores, making it well-suited for practical applications in simulating the time-evolving corona.

astro-ph.SR↗

arXiv2025

Efficient magnetohydrodynamic modelling of the time-evolving corona by COCONUT

Compared to quasi-steady-state corona models that are constrained by a time-invariant magnetogram over a CR period, time-evolving corona models driven by time-varying photospheric magnetograms are more realistic and can maintain more useful information to accurately describe solar wind evolution and forecast CME propagation. This paper demonstrate that time-evolving corona simulations can be performed efficiently and accurately using an implicit method with relatively large time steps. We also evaluate differences between coronal structures captured by time-evolving and quasi-steady simulations over a CR period during solar minimum. We used a series of hourly updated photospheric magnetograms to drive the evolution of coronal structures from the solar surface to $25\; R_s$ during two CRs around the 2019 eclipse in an inertial coordinate system. We compare the time-evolving and quasi-steady simulations to demonstrate that the differences in these two types of coronal modelling can be obvious even for a solar minimum. The relative differences in radial velocity and density can be over $15 \%$ and $25 \%$ at 20$\;R_s$ during one CR period. We also evaluated the impact of time steps on the simulation results. Using a time step of approximately 10 minutes balances efficiency and necessary numerical stability and accuracy for time-evolving corona simulations around solar minima, with coronal evolution during a full CR simulated within only 9 hours (using 1080 CPU cores for 1.5M grid cells). The simulation results demonstrate that time-evolving MHD coronal simulations can be performed efficiently and accurately using an implicit method, offering a more realistic alternative to quasi-steady-state simulations. The fully implicit time-evolving corona model thus promises to simulate the time-evolving corona accurately in practical space weather forecasting.

astro-ph.SR↗

arXiv2025

E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing

The exponential growth in model sizes has significantly increased the communication burden in Federated Learning (FL). Existing methods to alleviate this burden by transmitting compressed gradients often face high compression errors, which slow down the model's convergence. To simultaneously achieve high compression effectiveness and lower compression errors, we study the gradient compression problem from a novel perspective. Specifically, we propose a systematical algorithm termed Extended Single-Step Synthetic Features Compressing (E-3SFC), which consists of three sub-components, i.e., the Single-Step Synthetic Features Compressor (3SFC), a double-way compression algorithm, and a communication budget scheduler. First, we regard the process of gradient computation of a model as decompressing gradients from corresponding inputs, while the inverse process is considered as compressing the gradients. Based on this, we introduce a novel gradient compression method termed 3SFC, which utilizes the model itself as a decompressor, leveraging training priors such as model weights and objective functions. 3SFC compresses raw gradients into tiny synthetic features in a single-step simulation, incorporating error feedback to minimize overall compression errors. To further reduce communication overhead, 3SFC is extended to E-3SFC, allowing double-way compression and dynamic communication budget scheduling. Our theoretical analysis under both strongly convex and non-convex conditions demonstrates that 3SFC achieves linear and sub-linear convergence rates with aggregation noise. Extensive experiments across six datasets and six models reveal that 3SFC outperforms state-of-the-art methods by up to 13.4% while reducing communication costs by 111.6 times. These findings suggest that 3SFC can significantly enhance communication efficiency in FL without compromising model performance.

cs.LG↗

arXiv2025

Retrieval-Augmented TLAPS Proof Generation with Large Language Models

We present a novel approach to automated proof generation for the TLA+ Proof System (TLAPS) using Large Language Models (LLMs). Our method combines two key components: a sub-proof obligation generation phase that breaks down complex proof obligations into simpler sub-obligations, and a proof generation phase that leverages Retrieval-Augmented Generation with verified proof examples. We evaluate our approach using proof obligations from varying complexity levels of proof obligations, spanning from fundamental arithmetic properties to the properties of algorithms. Our experiments demonstrate that while the method successfully generates valid proofs for intermediate-complexity obligations, it faces limitations with more complex theorems. These results indicate that our approach can effectively assist in proof development for certain classes of properties, contributing to the broader goal of integrating LLMs into formal verification workflows.

cs.LO↗

arXiv2024

Frozen-field Modeling of Coronal Condensations with MPI-AMRVAC II: Optimization and application in three-dimensional models

The frozen-field hydrodynamic (ffHD) model is a simplification of the full magnetohydrodynamical (MHD) equations under the assumption of a rigid magnetic field, which significantly reduces computational complexity and enhances efficiency. In this work, we combine the ffHD prescription with hyperbolic thermal conduction (TC) and the Transition Region Adaptive Conduction (TRAC) method to achieve further optimization. A series of two-dimensional tests are done to evaluate the performance of the hyperbolic TC and the TRAC method. The results indicate that hyperbolic TC, while showing limiter-affected numerical dissipation, delivers outcomes comparable to classic parabolic TC. The TRAC method effectively compensates for the underestimation of enthalpy flux in low-resolution simulations, as evaluated on tests that demonstrate prominence formation. We present an application of the ffHD model that forms a three-dimensional prominence embedded in a magnetic flux rope, which develops into a stable slab-like filament. The simulation reveals a prominence with an elongated spine and a width consistent with observations, highlighting the potential of the ffHD model in capturing the dynamics of solar prominences. Forward modeling of the simulation data produces synthetic images at various wavelengths, providing insights into the appearance of prominences and filaments in different observational contexts. The ffHD model, with its computational efficiency and the demonstrated capability to simulate complex solar phenomena, offers a valuable tool for solar physicists, and is implemented in the open-source MPI-AMRVAC framework.

astro-ph.SR↗

arXiv2024

In-Hand Singulation and Scooping Manipulation with a 5 DOF Tactile Gripper

Manipulation tasks often require a high degree of dexterity, typically necessitating grippers with multiple degrees of freedom (DoF). While a robotic hand equipped with multiple fingers can execute precise and intricate manipulation tasks, the inherent redundancy stemming from its extensive DoF often adds unnecessary complexity. In this paper, we introduce the design of a tactile sensor-equipped gripper with two fingers and five DoF. We present a novel design integrating a GelSight tactile sensor, enhancing sensing capabilities and enabling finer control during specific manipulation tasks. To evaluate the gripper's performance, we conduct experiments involving two challenging tasks: 1) retrieving, singularizing, and classification of various objects embedded in granular media, and 2) executing scooping manipulations of credit cards in confined environments to achieve precise insertion. Our results demonstrate the efficiency of the proposed approach, with a high success rate for singulation and classification tasks, particularly for spherical objects at high as 94.3%, and a 100% success rate for scooping and inserting credit cards.

cs.RO↗

arXiv2024

Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equations

Bayesian flow networks (BFNs) iteratively refine the parameters, instead of the samples in diffusion models (DMs), of distributions at various noise levels through Bayesian inference. Owing to its differentiable nature, BFNs are promising in modeling both continuous and discrete data, while simultaneously maintaining fast sampling capabilities. This paper aims to understand and enhance BFNs by connecting them with DMs through stochastic differential equations (SDEs). We identify the linear SDEs corresponding to the noise-addition processes in BFNs, demonstrate that BFN's regression losses are aligned with denoise score matching, and validate the sampler in BFN as a first-order solver for the respective reverse-time SDE. Based on these findings and existing recipes of fast sampling in DMs, we propose specialized solvers for BFNs that markedly surpass the original BFN sampler in terms of sample quality with a limited number of function evaluations (e.g., 10) on both image and text datasets. Notably, our best sampler achieves an increase in speed of 5~20 times for free. Our code is available at https://github.com/ML-GSAI/BFN-Solver.

cs.LG↗

arXiv2024

CC2Vec: Combining Typed Tokens with Contrastive Learning for Effective Code Clone Detection

With the development of the open source community, the code is often copied, spread, and evolved in multiple software systems, which brings uncertainty and risk to the software system (e.g., bug propagation and copyright infringement). Therefore, it is important to conduct code clone detection to discover similar code pairs. Many approaches have been proposed to detect code clones where token-based tools can scale to big code. However, due to the lack of program details, they cannot handle more complicated code clones, i.e., semantic code clones. In this paper, we introduce CC2Vec, a novel code encoding method designed to swiftly identify simple code clones while also enhancing the capability for semantic code clone detection. To retain the program details between tokens, CC2Vec divides them into different categories (i.e., typed tokens) according to the syntactic types and then applies two self-attention mechanism layers to encode them. To resist changes in the code structure of semantic code clones, CC2Vec performs contrastive learning to reduce the differences introduced by different code implementations. We evaluate CC2Vec on two widely used datasets (i.e., BigCloneBench and Google Code Jam) and the results report that our method can effectively detect simple code clones. In addition, CC2Vec not only attains comparable performance to widely used semantic code clone detection systems such as ASTNN, SCDetector, and FCCA by simply fine-tuning, but also significantly surpasses these methods in both detection efficiency.

cs.SE↗

arXiv2024

Frozen-field Modeling of Coronal Condensations with MPI-AMRVAC I: Demonstration in two-dimensional models

Large-scale coronal plasma evolutions can be adequately described by magnetohydrodynamics (MHD) equations. However, full multi-dimensional MHD simulations require substantial computational resources. Given the low plasma $β$ in the solar corona, in many coronal studies, it suffices to approximate the magnetic field to remain topologically fixed and effectively conduct one-dimensional (1D) hydrodynamic (HD) simulations instead. This approach is often employed in studies of coronal loops and their liability to form condensations related to thermal instability. While 1D HD simulations along given and fixed field line shapes are convenient and fast, they are difficult to directly compare with multi-dimensional phenomena. Therefore, it is more convenient to solve volume-filling, multi-dimensional versions of the MHD equations where we freeze the magnetic field, transforming it into frozen-field HD (ffHD) equations for simulation. We have incorporated this ffHD module into our open-source MPI-AMRVAC code and tested it using a two-dimensional (2D) evaporation--condensation model to study prominence formation due to radiative losses. The 2D ffHD results are compared with those from actual 2D MHD and pseudo-2D HD simulations, analyzing the differences and their causes. Pseudo-2D studies account for the known {flux tube} expansion effects. Overall, the performance of 2D ffHD is close to that of 2D MHD and pseudo-2D HD. The 2D tests conducted in this paper will be extended in follow-up studies to 3D simulations based on analytical or observational approaches.

astro-ph.SR↗

arXiv2024

Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement

To enhance the multi-step reasoning capabilities of large language models, researchers have extensively explored prompting methods, notably the Chain-of-Thought (CoT) method which explicitly elicits human-like rationales. However, they have inadvertently overlooked the potential of enhancing model reasoning performance by formulating higher-quality problems. In this work, we start from the problem side and propose Self-Polish (SP), a novel method that facilitates the model's reasoning by guiding it to progressively refine the given problems to be more comprehensible and solvable. We also explore several automatic prompting varients and propose the Self-Polish prompt bank for the community. SP is orthogonal to all other prompting methods of answer/reasoning side like CoT, allowing for seamless integration with state-of-the-art techniques for further improvement. Thorough experiments show that the proposed method attains notable and consistent effectiveness on five reasoning benchmarks across different models. Furthermore, our method also showcases impressive performance on robustness evaluation. Codes and prompts are available at https://github.com/WooooDyy/Self-Polish.

cs.CL↗

arXiv2024

Gradient Estimation with Discrete Stein Operators

Gradient estimation -- approximating the gradient of an expectation with respect to the parameters of a distribution -- is central to the solution of many machine learning problems. However, when the distribution is discrete, most common gradient estimators suffer from excessive variance. To improve the quality of gradient estimation, we introduce a variance reduction technique based on Stein operators for discrete distributions. We then use this technique to build flexible control variates for the REINFORCE leave-one-out estimator. Our control variates can be adapted online to minimize variance and do not require extra evaluations of the target function. In benchmark generative modeling tasks such as training binary variational autoencoders, our gradient estimator achieves substantially lower variance than state-of-the-art estimators with the same number of function evaluations.

stat.ML↗

arXiv2024

Compositional Inductive Invariant Based Verification of Neural Network Controlled Systems

The integration of neural networks into safety-critical systems has shown great potential in recent years. However, the challenge of effectively verifying the safety of Neural Network Controlled Systems (NNCS) persists. This paper introduces a novel approach to NNCS safety verification, leveraging the inductive invariant method. Verifying the inductiveness of a candidate inductive invariant in the context of NNCS is hard because of the scale and nonlinearity of neural networks. Our compositional method makes this verification process manageable by decomposing the inductiveness proof obligation into smaller, more tractable subproblems. Alongside the high-level method, we present an algorithm capable of automatically verifying the inductiveness of given candidates by automatically inferring the necessary decomposition predicates. The algorithm significantly outperforms the baseline method and shows remarkable reductions in execution time in our case studies, shortening the verification time from hours (or timeout) to seconds.

cs.LO↗

arXiv2024

Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals

Deep neural networks (DNNs) are notoriously vulnerable to adversarial attacks that place carefully crafted perturbations on normal examples to fool DNNs. To better understand such attacks, a characterization of the features carried by adversarial examples is needed. In this paper, we tackle this challenge by inspecting the subspaces of sample features through spectral analysis. We first empirically show that the features of either clean signals or adversarial perturbations are redundant and span in low-dimensional linear subspaces respectively with minimal overlap, and the classical low-dimensional subspace projection can suppress perturbation features out of the subspace of clean signals. This makes it possible for DNNs to learn a subspace where only features of clean signals exist while those of perturbations are discarded, which can facilitate the distinction of adversarial examples. To prevent the residual perturbations that is inevitable in subspace learning, we propose an independence criterion to disentangle clean signals from perturbations. Experimental results show that the proposed strategy enables the model to inherently suppress adversaries, which not only boosts model robustness but also motivates new directions of effective adversarial defense.

cs.LG↗

arXiv2024

Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning

In this paper, we propose R$^3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for large language models. The core challenge in applying RL to complex reasoning is to identify a sequence of actions that result in positive rewards and provide appropriate supervision for optimization. Outcome supervision provides sparse rewards for final results without identifying error locations, whereas process supervision offers step-wise rewards but requires extensive manual annotation. R$^3$ overcomes these limitations by learning from correct demonstrations. Specifically, R$^3$ progressively slides the start state of reasoning from a demonstration's end to its beginning, facilitating easier model exploration at all stages. Thus, R$^3$ establishes a step-wise curriculum, allowing outcome supervision to offer step-level signals and precisely pinpoint errors. Using Llama2-7B, our method surpasses RL baseline on eight reasoning tasks by $4.1$ points on average. Notebaly, in program-based reasoning on GSM8K, it exceeds the baseline by $4.2$ points across three backbone models, and without any extra data, Codellama-7B + R$^3$ performs comparable to larger models or closed-source models.

cs.AI↗

arXiv2024

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

Supervised fine-tuning (SFT) is a crucial step for large language models (LLMs), enabling them to align with human instructions and enhance their capabilities in downstream tasks. Increasing instruction data substantially is a direct solution to align the model with a broader range of downstream tasks or notably improve its performance on a specific task. However, we find that large-scale increases in instruction data can damage the world knowledge previously stored in LLMs. To address this challenge, we propose LoRAMoE, a novelty framework that introduces several low-rank adapters (LoRA) and integrates them by using a router network, like a plugin version of Mixture of Experts (MoE). It freezes the backbone model and forces a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks, to alleviate world knowledge-edge forgetting. Experimental results show that, as the instruction data increases, LoRAMoE can significantly improve the ability to process downstream tasks, while maintaining the world knowledge stored in the LLM.

cs.CL↗

arXiv2024

Temporal Knowledge Graph Completion with Time-sensitive Relations in Hypercomplex Space

Temporal knowledge graph completion (TKGC) aims to fill in missing facts within a given temporal knowledge graph at a specific time. Existing methods, operating in real or complex spaces, have demonstrated promising performance in this task. This paper advances beyond conventional approaches by introducing more expressive quaternion representations for TKGC within hypercomplex space. Unlike existing quaternion-based methods, our study focuses on capturing time-sensitive relations rather than time-aware entities. Specifically, we model time-sensitive relations through time-aware rotation and periodic time translation, effectively capturing complex temporal variability. Furthermore, we theoretically demonstrate our method's capability to model symmetric, asymmetric, inverse, compositional, and evolutionary relation patterns. Comprehensive experiments on public datasets validate that our proposed approach achieves state-of-the-art performance in the field of TKGC.

cs.LG↗

arXiv2024

A Survey on Temporal Knowledge Graph: Representation Learning and Applications

Knowledge graphs have garnered significant research attention and are widely used to enhance downstream applications. However, most current studies mainly focus on static knowledge graphs, whose facts do not change with time, and disregard their dynamic evolution over time. As a result, temporal knowledge graphs have attracted more attention because a large amount of structured knowledge exists only within a specific period. Knowledge graph representation learning aims to learn low-dimensional vector embeddings for entities and relations in a knowledge graph. The representation learning of temporal knowledge graphs incorporates time information into the standard knowledge graph framework and can model the dynamics of entities and relations over time. In this paper, we conduct a comprehensive survey of temporal knowledge graph representation learning and its applications. We begin with an introduction to the definitions, datasets, and evaluation metrics for temporal knowledge graph representation learning. Next, we propose a taxonomy based on the core technologies of temporal knowledge graph representation learning methods, and provide an in-depth analysis of different methods in each category. Finally, we present various downstream applications related to the temporal knowledge graphs. In the end, we conclude the paper and have an outlook on the future research directions in this area.

cs.CL↗