Searcharxiv⌕ Search

arXiv subjects

Yuhao Zhou

Publications and source records attributed to Yuhao Zhou.

At least 109 records · Page 6Linked to original sources
arXiv2024

The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing

We present a unified probabilistic formulation for diffusion-based image editing, where a latent variable is edited in a task-specific manner and generally deviates from the corresponding marginal distribution induced by the original stochastic or ordinary differential equation (SDE or ODE). Instead, it defines a corresponding SDE or ODE for editing. In the formulation, we prove that the Kullback-Leibler divergence between the marginal distributions of the two SDEs gradually decreases while that for the ODEs remains as the time approaches zero, which shows the promise of SDE in image editing. Inspired by it, we provide the SDE counterparts for widely used ODE baselines in various tasks including inpainting and image-to-image translation, where SDE shows a consistent and substantial improvement. Moreover, we propose SDE-Drag -- a simple yet effective method built upon the SDE formulation for point-based content dragging. We build a challenging benchmark (termed DragBench) with open-set natural, art, and AI-generated images for evaluation. A user study on DragBench indicates that SDE-Drag significantly outperforms our ODE baseline, existing diffusion-based methods, and the renowned DragGAN. Our results demonstrate the superiority and versatility of SDE in image editing and push the boundary of diffusion-based editing methods.

cs.CV↗
arXiv2024

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

The advancement of large language models (LLMs) has significantly propelled the field of code generation. Previous work integrated reinforcement learning (RL) with compiler feedback for exploring the output space of LLMs to enhance code generation quality. However, the lengthy code generated by LLMs in response to complex human requirements makes RL exploration a challenge. Also, since the unit tests may not cover the complicated code, optimizing LLMs by using these unexecuted code snippets is ineffective. To tackle these challenges, we introduce StepCoder, a novel RL framework for code generation, consisting of two main components: CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks, while FGO only optimizes the model by masking the unexecuted code segments to provide Fine-Grained Optimization. In addition, we furthermore construct the APPS+ dataset for RL training, which is manually verified to ensure the correctness of unit tests. Experimental results show that our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks. Our dataset APPS+ and StepCoder are available online.

cs.SE↗
arXiv2024

MouSi: Poly-Visual-Expert Vision-Language Models

Current large vision-language models (VLMs) often encounter challenges such as insufficient capabilities of a single visual component and excessively long visual tokens. These issues can limit the model's effectiveness in accurately interpreting complex visual information and over-lengthy contextual information. Addressing these challenges is crucial for enhancing the performance and applicability of VLMs. This paper proposes the use of ensemble experts technique to synergizes the capabilities of individual visual encoders, including those skilled in image-text matching, OCR, image segmentation, etc. This technique introduces a fusion network to unify the processing of outputs from different visual experts, while bridging the gap between image encoders and pre-trained LLMs. In addition, we explore different positional encoding schemes to alleviate the waste of positional encoding caused by lengthy image feature sequences, effectively addressing the issue of position overflow and length limitations. For instance, in our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient and manageable 64 or even down to 1. Experimental results demonstrate that VLMs with multiple experts exhibit consistently superior performance over isolated visual encoders and mark a significant performance boost as more experts are integrated. We have open-sourced the training code used in this report. All of these resources can be found on our project website.

cs.CV↗
arXiv2024

Secrets of RLHF in Large Language Models Part II: Reward Modeling

Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive reinforcement learning optimization. While reward models are often considered central to achieving high performance, they face the following challenges in practical applications: (1) Incorrect and ambiguous preference pairs in the dataset may hinder the reward model from accurately capturing human intent. (2) Reward models trained on data from a specific distribution often struggle to generalize to examples outside that distribution and are not suitable for iterative RLHF training. In this report, we attempt to address these two issues. (1) From a data perspective, we propose a method to measure the strength of preferences within the data, based on a voting mechanism of multiple reward models. Experimental results confirm that data with varying preference strengths have different impacts on reward model performance. We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset and fully leverage high-quality preference data. (2) From an algorithmic standpoint, we introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses, thereby improving model generalization. Furthermore, we employ meta-learning to enable the reward model to maintain the ability to differentiate subtle differences in out-of-distribution samples, and this approach can be utilized for iterative RLHF optimization.

cs.AI↗
arXiv2023

Improving Generalization of Alignment with Human Preferences through Group Invariant Learning

The success of AI assistants based on language models (LLMs) hinges crucially on Reinforcement Learning from Human Feedback (RLHF), which enables the generation of responses more aligned with human preferences. As universal AI assistants, there's a growing expectation for them to perform consistently across various domains. However, previous work shows that Reinforcement Learning (RL) often exploits shortcuts to attain high rewards and overlooks challenging samples. This focus on quick reward gains undermines both the stability in training and the model's ability to generalize to new, unseen data. In this work, we propose a novel approach that can learn a consistent policy via RL across various data groups or domains. Given the challenges associated with acquiring group annotations, our method automatically classifies data into different groups, deliberately maximizing performance variance. Then, we optimize the policy to perform well on challenging groups. Lastly, leveraging the established groups, our approach adaptively adjusts the exploration space, allocating more learning capacity to more challenging data and preventing the model from over-optimizing on simpler data. Experimental results indicate that our approach significantly enhances training stability and model generalization.

cs.LG↗
arXiv2023

PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning

Classical federated learning (FL) enables training machine learning models without sharing data for privacy preservation, but heterogeneous data characteristic degrades the performance of the localized model. Personalized FL (PFL) addresses this by synthesizing personalized models from a global model via training on local data. Such a global model may overlook the specific information that the clients have been sampled. In this paper, we propose a novel scheme to inject personalized prior knowledge into the global model in each client, which attempts to mitigate the introduced incomplete information problem in PFL. At the heart of our proposed approach is a framework, the PFL with Bregman Divergence (pFedBreD), decoupling the personalized prior from the local objective function regularized by Bregman divergence for greater adaptability in personalized scenarios. We also relax the mirror descent (RMD) to extract the prior explicitly to provide optional strategies. Additionally, our pFedBreD is backed up by a convergence analysis. Sufficient experiments demonstrate that our method reaches the state-of-the-art performances on 5 datasets and outperforms other methods by up to 3.5% across 8 benchmarks. Extensive analyses verify the robustness and necessity of proposed designs.

cs.LG↗
arXiv2023

The Rise and Potential of Large Language Model Based Agents: A Survey

For a long time, humanity has pursued artificial intelligence (AI) equivalent to or surpassing the human level, with AI agents considered a promising vehicle for this pursuit. AI agents are artificial entities that sense their environment, make decisions, and take actions. Many efforts have been made to develop intelligent agents, but they mainly focus on advancement in algorithms or training strategies to enhance specific capabilities or performance on particular tasks. Actually, what the community lacks is a general and powerful model to serve as a starting point for designing AI agents that can adapt to diverse scenarios. Due to the versatile capabilities they demonstrate, large language models (LLMs) are regarded as potential sparks for Artificial General Intelligence (AGI), offering hope for building general AI agents. Many researchers have leveraged LLMs as the foundation to build AI agents and have achieved significant progress. In this paper, we perform a comprehensive survey on LLM-based agents. We start by tracing the concept of agents from its philosophical origins to its development in AI, and explain why LLMs are suitable foundations for agents. Building upon this, we present a general framework for LLM-based agents, comprising three main components: brain, perception, and action, and the framework can be tailored for different applications. Subsequently, we explore the extensive applications of LLM-based agents in three aspects: single-agent scenarios, multi-agent scenarios, and human-agent cooperation. Following this, we delve into agent societies, exploring the behavior and personality of LLM-based agents, the social phenomena that emerge from an agent society, and the insights they offer for human society. Finally, we discuss several key topics and open problems within the field. A repository for the related papers at https://github.com/WooooDyy/LLM-Agent-Paper-List.

cs.AI↗
arXiv2023

Federated cINN Clustering for Accurate Clustered Federated Learning

Federated Learning (FL) presents an innovative approach to privacy-preserving distributed machine learning and enables efficient crowd intelligence on a large scale. However, a significant challenge arises when coordinating FL with crowd intelligence which diverse client groups possess disparate objectives due to data heterogeneity or distinct tasks. To address this challenge, we propose the Federated cINN Clustering Algorithm (FCCA) to robustly cluster clients into different groups, avoiding mutual interference between clients with data heterogeneity, and thereby enhancing the performance of the global model. Specifically, FCCA utilizes a global encoder to transform each client's private data into multivariate Gaussian distributions. It then employs a generative model to learn encoded latent features through maximum likelihood estimation, which eases optimization and avoids mode collapse. Finally, the central server collects converged local models to approximate similarities between clients and thus partition them into distinct clusters. Extensive experimental results demonstrate FCCA's superiority over other state-of-the-art clustered federated learning algorithms, evaluated on various models and datasets. These results suggest that our approach has substantial potential to enhance the efficiency and accuracy of real-world federated learning tasks.

cs.DC↗
arXiv2023

Secrets of RLHF in Large Language Models Part I: PPO

Large language models (LLMs) have formulated a blueprint for the advancement of artificial general intelligence. Its primary objective is to function as a human-centric (helpful, honest, and harmless) assistant. Alignment with humans assumes paramount significance, and reinforcement learning with human feedback (RLHF) emerges as the pivotal technological paradigm underpinning this pursuit. Current technical routes usually include \textbf{reward models} to measure human preferences, \textbf{Proximal Policy Optimization} (PPO) to optimize policy model outputs, and \textbf{process supervision} to improve step-by-step reasoning capabilities. However, due to the challenges of reward design, environment interaction, and agent training, coupled with huge trial and error cost of large language models, there is a significant barrier for AI researchers to motivate the development of technical alignment and safe landing of LLMs. The stable training of RLHF has still been a puzzle. In the first report, we dissect the framework of RLHF, re-evaluate the inner workings of PPO, and explore how the parts comprising PPO algorithms impact policy agent training. We identify policy constraints being the key factor for the effective implementation of the PPO algorithm. Therefore, we explore the PPO-max, an advanced version of PPO algorithm, to efficiently improve the training stability of the policy model. Based on our main results, we perform a comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT. The absence of open-source implementations has posed significant challenges to the investigation of LLMs alignment. Therefore, we are eager to release technical reports, reward models and PPO codes, aiming to make modest contributions to the advancement of LLMs.

cs.CL↗
arXiv2023

Winking filaments due to cyclic evaporation-condensation

Observations have shown that some filaments appear and disappear in the H$α$ line wing images periodically. There have been no attempts to model these "winking filaments" thus far. The evaporation--condensation mechanism is widely used to explain the formation of solar filaments. Here, we demonstrate, for the first time, how multi-dimensional evaporation--condensation in an arcade setup invariably causes a stretching of the magnetic topology. We aim to check whether this magnetic stretching during cyclic evaporation--condensation could reproduce a winking filament. We used our open-source code MPI-AMRVAC to carry out 2D magnetohydrodynamic simulations based on a quadrupolar configuration. A periodic localized heating, which modulates the evaporation--condensation process, was imposed before, during, and after the formation of the filament. Synthetic H$α$ and 304 Å, images were produced to compare the results with observations. For the first time, we noticed the winking filament phenomenon in a simulation of the formation of on-disk solar filaments, which was in good agreement with observations. Typically, the period of the winking is different from the period of the impulsive heating. A forced oscillator model explains this difference and fits the results well. A parameter survey is also done to look into details of the magnetic stretching phenomenon. We found that the stronger the heating or the higher the layer where the heating occurs, the more significant the winking effect appears.

astro-ph.SR↗
arXiv2023

Transverse vertical oscillations during the contraction and expansion of coronal loops

In this paper, we carry out a detailed analysis of the M1.6 class eruptive flare occurring in NOAA active region 13078 on 2022 August 19. The flare is associated with a fast coronal mass ejection (CME) propagating in the southwest direction with an apparent speed of $\sim$926 km s$^{-1}$. Meanwhile, a shock wave is driven by the CME at the flank. The eruption of CME generates an extreme-ultraviolet (EUV) wave expanding outward from the flare site with an apparent speed of $\geq$200 km s$^{-1}$. As the EUV wave propagates eastward, it encounters and interacts with the low-lying adjacent coronal loops (ACLs), which are composed of two loops. The compression of EUV wave results in contraction, expansion, and transverse vertical oscillations of ACLs. The commencements of contraction are sequential from western to eastern footpoints and the contraction lasts for $\sim$15 minutes. The speeds of contraction lie in the range of 13$-$40 km s$^{-1}$ in 171 Å and 8$-$54 km s$^{-1}$ in 193 Å. A long, gradual expansion follows the contraction at lower speeds. Concurrent vertical oscillations are superposed on contraction and expansion of ACLs. The oscillations last for 2$-$9 cycles and the amplitudes are $\leq$4 Mm. The periods are between 3 to 12 minutes with an average value of 6.7 minutes. The results show rich dynamics of coronal loops.

astro-ph.SR↗
arXiv2023

Multi-thermal jet formation triggered by flux emergence

Flux emergence is responsible for various solar eruptions. Combining observation and simulations, we investigate the influence of flux emergence at one footpoint of an arcade on coronal rain as well as induced eruptions. The emergence changes the pressure in the loops, and the internal coronal rain all moves to the other side. The emerging flux reconnects with the overlying magnetic field, forming a current sheet and magnetic islands. The plasma is ejected outwards and heated, forming a cool jet ~ 6000 K and a hot X-ray jet ~ 4 MK simultaneously. The jet dynamical properties agree very well between observation and simulation. In the simulation, the jet also displays transverse oscillations with a period of 8 minutes, a so-called whip-like motion. The movement of the jet and dense plasmoids changes the configuration of the local magnetic field, facilitating the occurrence of Kelvin--Helmholtz instability, and vortex-like structures form at the boundary of the jet. Our simulation clearly demonstrates the effect of emergence on coronal rain, the dynamical details of reconnecting plasmoid chains, the formation of multi-thermal jets, and the cycling of cool mass between the chromosphere and the corona.

astro-ph.SR↗
arXiv2023

Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence

Reducing communication overhead in federated learning (FL) is challenging but crucial for large-scale distributed privacy-preserving machine learning. While methods utilizing sparsification or others can largely lower the communication overhead, the convergence rate is also greatly compromised. In this paper, we propose a novel method, named single-step synthetic features compressor (3SFC), to achieve communication-efficient FL by directly constructing a tiny synthetic dataset based on raw gradients. Thus, 3SFC can achieve an extremely low compression rate when the constructed dataset contains only one data sample. Moreover, 3SFC's compressing phase utilizes a similarity-based objective function so that it can be optimized with just one step, thereby considerably improving its performance and robustness. In addition, to minimize the compressing error, error feedback (EF) is also incorporated into 3SFC. Experiments on multiple datasets and models suggest that 3SFC owns significantly better convergence rates compared to competing methods with lower compression rates (up to 0.02%). Furthermore, ablation studies and visualizations show that 3SFC can carry more information than competing methods for every communication round, further validating its effectiveness.

cs.LG↗
arXiv2023

MPI-AMRVAC 3.0: updates to an open-source simulation framework

Computational astrophysics routinely combines grid-adaptive capabilities with modern shock-capturing, high resolution spatio-temporal schemes on multi-dimensional hydro- and magnetohydrodynamics. We provide an update on developments within the open-source MPI-AMRVAC code. With online documentation, the MPI-AMRVAC 3.0 release includes several added equation sets, and many options to explore and quantify the influence of implementation details. Showcasing this on a variety of hydro and MHD tests, we document new modules of interest for state-of-the-art solar applications. Test cases address how higher order reconstructions impact long term simulations of shear layers, with and without gas-dust coupling, how runaway radiative losses transit to intricate multi-temperature, multi-phase dynamics, and how different flavors of spatio-temporal schemes and magnetic monopole control produce consistent MHD results in combination with adaptive meshes. We demonstrate Super-Time-Stepping strategies for specific parabolic terms and give details on all implemented Implicit-Explicit integrators. A new magnetofrictional module can be used for computing force-free magnetic fields or for data-driven time-dependent evolutions, while the Regularized-Biot-Savart-Law approach can insert fluxropes in 3D domains. Synthetic observations of 3D MHD simulations can be rendered on-the-fly, or in post-processing, in many spectral wavebands. A particle module and a generic fieldline tracing, compatible with the hierarchical meshes, can be used to sample information at prescribed locations, to follow dynamics of charged particles, or realize two-way coupled simulations between MHD setups and field-aligned non-thermal processes. Highlighting the latest additions and various technical aspects, our open-source strategy welcomes any further code usage, contribution, or spin-off development.

astro-ph.IM↗
arXiv2023

Rapid Rotation of an Erupting Prominence and the Associated Coronal Mass Ejection on 13 May 2013

In this paper, we report the multiwavelength observations of an erupting prominence and the associated CME on 13 May 2013. The event occurs behind the western limb in the field of view of SDO/AIA. The prominence is supported by a highly twisted magnetic flux rope and shows rapid rotation in the counterclockwise direction during the rising motion. The rotation of the prominence lasts for $\sim$47 minutes. The average period, angular speed, and linear speed are $\sim$806 s, $\sim$0.46 rad min$^{-1}$, and $\sim$355 km s$^{-1}$, respectively. The total twist angle reaches $\sim$7$π$, which is considerably larger than the threshold for kink instability. Writhing motion during 17:42$-$17:46 UT is clearly observed by SWAP in 174 Å and EUVI on board the behind STEREO spacecraft in 304 Å after reaching an apparent height of $\sim$405\,Mm. Therefore, the prominence eruption is most probably triggered by kink instability. A pair of conjugate flare ribbons and post-flare loops are created and observed by STA/EUVI. The onset time of writhing motion is consistent with the commencement of the impulsive phase of the related flare. The 3D morphology and positions of the associated CME are derived using the graduated cylindrical shell (GCS) modeling. The kinetic evolution of the reconstructed CME is divided into a slow-rise phase ($\sim$330 km s$^{-1}$) and a fast-rise phase ($\sim$1005 km s$^{-1}$) by the writhing motion. The edge-on angular width of the CME is a constant (60$^{\circ}$), while the face-on angular width increases from 96$^{\circ}$ to 114$^{\circ}$, indicating a lateral expansion. The latitude of the CME source region decreases slightly from $\sim$18$^{\circ}$ to $\sim$13$^{\circ}$, implying an equatorward deflection during propagation.

astro-ph.SR↗
arXiv2022

Personalized Federated Learning with Hidden Information on Personalized Prior

Federated learning (FL for simplification) is a distributed machine learning technique that utilizes global servers and collaborative clients to achieve privacy-preserving global model training without direct data sharing. However, heterogeneous data problem, as one of FL's main problems, makes it difficult for the global model to perform effectively on each client's local data. Thus, personalized federated learning (PFL for simplification) aims to improve the performance of the model on local data as much as possible. Bayesian learning, where the parameters of the model are seen as random variables with a prior assumption, is a feasible solution to the heterogeneous data problem due to the tendency that the more local data the model use, the more it focuses on the local data, otherwise focuses on the prior. When Bayesian learning is applied to PFL, the global model provides global knowledge as a prior to the local training process. In this paper, we employ Bayesian learning to model PFL by assuming a prior in the scaled exponential family, and therefore propose pFedBreD, a framework to solve the problem we model using Bregman divergence regularization. Empirically, our experiments show that, under the prior assumption of the spherical Gaussian and the first order strategy of mean selection, our proposal significantly outcompetes other PFL algorithms on multiple public benchmarks.

cs.LG↗
arXiv2022

Robust Lottery Tickets for Pre-trained Language Models

Recent works on Lottery Ticket Hypothesis have shown that pre-trained language models (PLMs) contain smaller matching subnetworks(winning tickets) which are capable of reaching accuracy comparable to the original models. However, these tickets are proved to be notrobust to adversarial examples, and even worse than their PLM counterparts. To address this problem, we propose a novel method based on learning binary weight masks to identify robust tickets hidden in the original PLMs. Since the loss is not differentiable for the binary mask, we assign the hard concrete distribution to the masks and encourage their sparsity using a smoothing approximation of L0 regularization.Furthermore, we design an adversarial loss objective to guide the search for robust tickets and ensure that the tickets perform well bothin accuracy and robustness. Experimental results show the significant improvement of the proposed method over previous work on adversarial robustness evaluation.

cs.CL↗
arXiv2022

DBS: Dynamic Batch Size For Distributed Deep Neural Network Training

Synchronous strategies with data parallelism, such as the Synchronous StochasticGradient Descent (S-SGD) and the model averaging methods, are widely utilizedin distributed training of Deep Neural Networks (DNNs), largely owing to itseasy implementation yet promising performance. Particularly, each worker ofthe cluster hosts a copy of the DNN and an evenly divided share of the datasetwith the fixed mini-batch size, to keep the training of DNNs convergence. In thestrategies, the workers with different computational capability, need to wait foreach other because of the synchronization and delays in network transmission,which will inevitably result in the high-performance workers wasting computation.Consequently, the utilization of the cluster is relatively low. To alleviate thisissue, we propose the Dynamic Batch Size (DBS) strategy for the distributedtraining of DNNs. Specifically, the performance of each worker is evaluatedfirst based on the fact in the previous epoch, and then the batch size and datasetpartition are dynamically adjusted in consideration of the current performanceof the worker, thereby improving the utilization of the cluster. To verify theeffectiveness of the proposed strategy, extensive experiments have been conducted,and the experimental results indicate that the proposed strategy can fully utilizethe performance of the cluster, reduce the training time, and have good robustnesswith disturbance by irrelevant tasks. Furthermore, rigorous theoretical analysis hasalso been provided to prove the convergence of the proposed strategy.

cs.LG↗