arXiv subjects

Fei Ye

Publications and source records attributed to Fei Ye.

At least 37 records · Page 2Linked to original sources

arXiv2025

Elucidating the Design Space of Multimodal Protein Language Models

Multimodal protein language models (PLMs) integrate sequence and token-based structural information, serving as a powerful foundation for protein modeling, generation, and design. However, the reliance on tokenizing 3D structures into discrete tokens causes substantial loss of fidelity about fine-grained structural details and correlations. In this paper, we systematically elucidate the design space of multimodal PLMs to overcome their limitations. We identify tokenization loss and inaccurate structure token predictions by the PLMs as major bottlenecks. To address these, our proposed design space covers improved generative modeling, structure-aware architectures and representation learning, and data exploration. Our advancements approach finer-grained supervision, demonstrating that token-based multimodal PLMs can achieve robust structural modeling. The effective design methods dramatically improve the structure generation diversity, and notably, folding abilities of our 650M model by reducing the RMSD from 5.52 to 2.36 on PDB testset, even outperforming 3B baselines and on par with the specialized folding models. Project page and code: https://bytedance.github.io/dplm/dplm-2.1/.

cs.LG↗

arXiv2025

Incrementally Learning Multiple Diverse Data Domains via Multi-Source Dynamic Expansion Model

Continual Learning seeks to develop a model capable of incrementally assimilating new information while retaining prior knowledge. However, current research predominantly addresses a straightforward learning context, wherein all data samples originate from a singular data domain. This paper shifts focus to a more complex and realistic learning environment, characterized by data samples sourced from multiple distinct domains. We tackle this intricate learning challenge by introducing a novel methodology, termed the Multi-Source Dynamic Expansion Model (MSDEM), which leverages various pre-trained models as backbones and progressively establishes new experts based on them to adapt to emerging tasks. Additionally, we propose an innovative dynamic expandable attention mechanism designed to selectively harness knowledge from multiple backbones, thereby accelerating the new task learning. Moreover, we introduce a dynamic graph weight router that strategically reuses all previously acquired parameters and representations for new task learning, maximizing the positive knowledge transfer effect, which further improves generalization performance. We conduct a comprehensive series of experiments, and the empirical findings indicate that our proposed approach achieves state-of-the-art performance.

cs.LG↗

arXiv2025

Self-Controlled Dynamic Expansion Model for Continual Learning

Continual Learning (CL) epitomizes an advanced training paradigm wherein prior data samples remain inaccessible during the acquisition of new tasks. Numerous investigations have delved into leveraging a pre-trained Vision Transformer (ViT) to enhance model efficacy in continual learning. Nonetheless, these approaches typically utilize a singular, static backbone, which inadequately adapts to novel tasks, particularly when engaging with diverse data domains, due to a substantial number of inactive parameters. This paper addresses this limitation by introducing an innovative Self-Controlled Dynamic Expansion Model (SCDEM), which orchestrates multiple distinct trainable pre-trained ViT backbones to furnish diverse and semantically enriched representations. Specifically, by employing the multi-backbone architecture as a shared module, the proposed SCDEM dynamically generates a new expert with minimal parameters to accommodate a new task. A novel Collaborative Optimization Mechanism (COM) is introduced to synergistically optimize multiple backbones by harnessing prediction signals from historical experts, thereby facilitating new task learning without erasing previously acquired knowledge. Additionally, a novel Feature Distribution Consistency (FDC) approach is proposed to align semantic similarity between previously and currently learned representations through an optimal transport distance-based mechanism, effectively mitigating negative knowledge transfer effects. Furthermore, to alleviate over-regularization challenges, this paper presents a novel Dynamic Layer-Wise Feature Attention Mechanism (DLWFAM) to autonomously determine the penalization intensity on each trainable representation layer. An extensive series of experiments have been conducted to evaluate the proposed methodology's efficacy, with empirical results corroborating that the approach attains state-of-the-art performance.

cs.LG↗

arXiv2025

On the Structure of Second Jacobian Ideals

We show that the second Jacobian ideal of a hypersurface can be decomposed such that a power of the Jacobian ideal becomes a factor. As an application of the decomposition, we present an elementary proof establishing that the second Nash blow-up algebra of a hypersurface singularity is a contact invariant.

math.AG↗

arXiv2025

Simulating Bulk Gap in Chiral Projected Entangled-Pair States

Projected entangled-pair states (PEPS) have proven effective in capturing chiral spin liquid ground states, yet the presence of long-range ``gossamer'' correlation tails raises concerns about their ability to accurately describe bulk gaps. Here, we address this challenge and demonstrate that PEPS can reliably characterize gapped bulk excitations in chiral topological phases. Using a variational principle for excited states within a local mode approximation, we establish that correlation functions decaying faster than $r^{-2}$ are not necessarily related to gapless modes and thus long-range ``gossamer'' correlation tails in chiral PEPS do not contradict the presence of a bulk gap. This framework is validated in the spin-$\frac{1}{2}$ Kitaev model with a chiral term, where PEPS yields excitation gaps that agree well with exact solutions. Extending our approach to the $\mathbb{Z}_3$ Kitaev model, we present compelling evidence for its chiral ground state and accurately resolve its gapped excitations. These findings thus solidify PEPS as a powerful tool for studying both ground and excited states in chiral topological systems, thereby bridging a key gap in the understanding of their bulk properties.

cond-mat.str-el↗

arXiv2025

Information-Theoretic Dual Memory System for Continual Learning

Continuously acquiring new knowledge from a dynamic environment is a fundamental capability for animals, facilitating their survival and ability to address various challenges. This capability is referred to as continual learning, which focuses on the ability to learn a sequence of tasks without the detriment of previous knowledge. A prevalent strategy to tackle continual learning involves selecting and storing numerous essential data samples from prior tasks within a fixed-size memory buffer. However, the majority of current memory-based techniques typically utilize a single memory buffer, which poses challenges in concurrently managing newly acquired and previously learned samples. Drawing inspiration from the Complementary Learning Systems (CLS) theory, which defines rapid and gradual learning mechanisms for processing information, we propose an innovative dual memory system called the Information-Theoretic Dual Memory System (ITDMS). This system comprises a fast memory buffer designed to retain temporary and novel samples, alongside a slow memory buffer dedicated to preserving critical and informative samples. The fast memory buffer is optimized employing an efficient reservoir sampling process. Furthermore, we introduce a novel information-theoretic memory optimization strategy that selectively identifies and retains diverse and informative data samples for the slow memory buffer. Additionally, we propose a novel balanced sample selection procedure that automatically identifies and eliminates redundant memorized samples, thus freeing up memory capacity for new data acquisitions, which can deal with a growing array of tasks. Our methodology is rigorously assessed through a series of continual learning experiments, with empirical results underscoring the effectiveness of the proposed system.

cs.LG↗

arXiv2024

Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesis

This paper introduces innovative solutions to enhance spatial controllability in diffusion models reliant on text queries. We first introduce vision guidance as a foundational spatial cue within the perturbed distribution. This significantly refines the search space in a zero-shot paradigm to focus on the image sampling process adhering to the spatial layout conditions. To precisely control the spatial layouts of multiple visual concepts with the employment of vision guidance, we propose a universal framework, Layered Rendering Diffusion (LRDiff), which constructs an image-rendering process with multiple layers, each of which applies the vision guidance to instructively estimate the denoising direction for a single object. Such a layered rendering strategy effectively prevents issues like unintended conceptual blending or mismatches while allowing for more coherent and contextually accurate image synthesis. The proposed method offers a more efficient and accurate means of synthesising images that align with specific layout and contextual requirements. Through experiments, we demonstrate that our method outperforms existing techniques, both quantitatively and qualitatively, in two specific layout-to-image tasks: bounding box-to-image and instance maskto-image. Furthermore, we extend the proposed framework to enable spatially controllable editing

cs.CV↗

arXiv2024

ProteinWeaver: A Divide-and-Assembly Approach for Protein Backbone Design

Nature creates diverse proteins through a 'divide and assembly' strategy. Inspired by this idea, we introduce ProteinWeaver, a two-stage framework for protein backbone design. Our method first generates individual protein domains and then employs an SE(3) diffusion model to flexibly assemble these domains. A key challenge lies in the assembling step, given the complex and rugged nature of the inter-domain interaction landscape. To address this challenge, we employ preference alignment to discern complex relationships between structure and interaction landscapes through comparative analysis of generated samples. Comprehensive experiments demonstrate that ProteinWeaver: (1) generates high-quality, novel protein backbones through versatile domain assembly; (2) outperforms RFdiffusion, the current state-of-the-art in backbone design, by 13\% and 39\% for long-chain proteins; (3) shows the potential for cooperative function design through illustrative case studies. To sum up, by introducing a `divide-and-assembly' paradigm, ProteinWeaver advances protein engineering and opens new avenues for functional protein design.

q-bio.BM↗

arXiv2024

DPLM-2: A Multimodal Diffusion Protein Language Model

Proteins are essential macromolecules defined by their amino acid sequences, which determine their three-dimensional structures and, consequently, their functions in all living organisms. Therefore, generative protein modeling necessitates a multimodal approach to simultaneously model, understand, and generate both sequences and structures. However, existing methods typically use separate models for each modality, limiting their ability to capture the intricate relationships between sequence and structure. This results in suboptimal performance in tasks that requires joint understanding and generation of both modalities. In this paper, we introduce DPLM-2, a multimodal protein foundation model that extends discrete diffusion protein language model (DPLM) to accommodate both sequences and structures. To enable structural learning with the language model, 3D coordinates are converted to discrete tokens using a lookup-free quantization-based tokenizer. By training on both experimental and high-quality synthetic structures, DPLM-2 learns the joint distribution of sequence and structure, as well as their marginals and conditionals. We also implement an efficient warm-up strategy to exploit the connection between large-scale evolutionary data and structural inductive biases from pre-trained sequence-based protein language models. Empirical evaluation shows that DPLM-2 can simultaneously generate highly compatible amino acid sequences and their corresponding 3D structures eliminating the need for a two-stage generation approach. Moreover, DPLM-2 demonstrates competitive performance in various conditional generation tasks, including folding, inverse folding, and scaffolding with multimodal motif inputs, as well as providing structure-aware representations for predictive tasks.

cs.LG↗

arXiv2024

Diffusion Language Models Are Versatile Protein Learners

This paper introduces diffusion protein language model (DPLM), a versatile protein language model that demonstrates strong generative and predictive capabilities for protein sequences. We first pre-train scalable DPLMs from evolutionary-scale protein sequences within a generative self-supervised discrete diffusion probabilistic framework, which generalizes language modeling for proteins in a principled way. After pre-training, DPLM exhibits the ability to generate structurally plausible, novel, and diverse protein sequences for unconditional generation. We further demonstrate the proposed diffusion generative pre-training makes DPLM possess a better understanding of proteins, making it a superior representation learner, which can be fine-tuned for various predictive tasks, comparing favorably to ESM2 (Lin et al., 2022). Moreover, DPLM can be tailored for various needs, which showcases its prowess of conditional generation in several ways: (1) conditioning on partial peptide sequences, e.g., generating scaffolds for functional motifs with high success rate; (2) incorporating other modalities as conditioner, e.g., structure-conditioned generation for inverse folding; and (3) steering sequence generation towards desired properties, e.g., satisfying specified secondary structures, through a plug-and-play classifier guidance. Code is released at \url{https://github.com/bytedance/dplm}.

cs.LG↗

arXiv2024

Muon Spin Relaxation Study of Spin Dynamics on a Kitaev honeycomb material H$_3$LiIr$_2$O$_6$

The vacancy effect in quantum spin liquid (QSL) has been extensively studied. A finite density of random vacancies in the Kitaev model can lead to a pileup of low-energy density of states (DOS), which is generally experimentally determined by a scaling behavior of thermodynamic or magnetization quantities. Here, we report detailed muon spin relaxation ($μ$SR) results of H$_3$LiIr$_2$O$_6$, a Kitaev QSL candidate with vacancies. The absence of magnetic order is confirmed down to 80 mK, and the spin fluctuations are found to be persistent at low temperatures. Intriguingly, the time-field scaling law of longitudinal-field (LF)-$μ$SR polarization is observed down to 0.1 K. This indicates a dynamical scaling, whose critical exponent 0.46 is excellently consistent with the scaling behavior of specific heat and magnetization data. All the observations point to the finite DOS with the form $N(E) \sim E^{-0.5}$ , which is expected for the Kitaev QSL in the presence of vacancies. Our μSR study provides a dynamical fingerprint of the power-law low-energy DOS, and introduces a crucial new insight into the vacancy effect in QSL.

cond-mat.str-el↗

arXiv2024

ProteinBench: A Holistic Evaluation of Protein Foundation Models

Recent years have witnessed a surge in the development of protein foundation models, significantly improving performance in protein prediction and generative tasks ranging from 3D structure prediction and protein design to conformational dynamics. However, the capabilities and limitations associated with these models remain poorly understood due to the absence of a unified evaluation framework. To fill this gap, we introduce ProteinBench, a holistic evaluation framework designed to enhance the transparency of protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance. Our comprehensive evaluation of protein foundation models reveals several key findings that shed light on their current capabilities and limitations. To promote transparency and facilitate further research, we release the evaluation dataset, code, and a public leaderboard publicly for further analysis and a general modular toolkit. We intend for ProteinBench to be a living benchmark for establishing a standardized, in-depth evaluation framework for protein foundation models, driving their development and application while fostering collaboration within the field.

q-bio.QM↗

arXiv2024

1-Form Symmetric Projected Entangled-Pair States

The 1-form symmetry, manifesting as loop-like symmetries, has gained prominence in the study of quantum phases, deepening our understanding of symmetry. However, the role of 1-form symmetries in Projected Entangled-Pair States (PEPS), two-dimensional tensor network states, remains largely underexplored. We present a novel framework for understanding 1-form symmetries within tensor networks, specifically focusing on the derivation of algebraic relations for symmetry matrices on the PEPS virtual legs. Our results reveal that 1-form symmetries impose stringent constraints on tensor network representations, leading to distinct anomalous braiding phases carried by symmetry matrices. We demonstrate how these symmetries influence the ground state and tangent space in PEPS, providing new insights into their physical implications for enhancing ground state optimization efficiency and characterizing the 1-form symmetry structure in excited states.

cond-mat.str-el↗

arXiv2024

Dynamic Multi-Objective Lion Swarm Optimization with Multi-strategy Fusion: An application in 6R robot trajectory planning

The advancement of industrialization has spurred the development of innovative swarm intelligence algorithms, with Lion Swarm Optimization (LSO) notable for its robustness, parallelism, simplicity, and efficiency. While LSO excels in single-objective optimization, its multi-objective variants face challenges such as poor initialization, local optima entrapment, and so on. This study proposes Dynamic Multi-Objective Lion Swarm Optimization with Multi-strategy Fusion (MF-DMOLSO) to address these limitations. MF-DMOLSO comprises three key components: initialization, swarm position update, and external archive update. The initialization unit employs chaotic mapping for uniform population distribution. The position update unit enhances behavior patterns and step size formulas for cub lions, incorporating crowding degree sorting, Pareto non-dominated sorting, and Levy flight to improve convergence speed and global search capabilities. Reference points guide convergence in higher-dimensional spaces, maintaining population diversity. An adaptive cold-hot start strategy generates a population responsive to environmental changes. The external archive update unit re-evaluates solutions based on non-domination and diversity to form the new population. Evaluations on benchmark functions showed MF-DMOLSO surpassed multi-objective particle swarm optimization, non-dominated sorting genetic algorithm II, and multi-objective lion swarm optimization, exceeding 90% accuracy for two-objective and 97% for three-objective problems. Compared to non-dominated sorting genetic algorithm III, MF-DMOLSO showed a 60% improvement. Applied to 6R robot trajectory planning, MF-DMOLSO optimized running time and maximum acceleration to 8.3s and 0.3pi rad/s^2, achieving a set coverage rate of 70.97% compared to 2% by multi-objective particle swarm optimization, thus improving efficiency and reducing mechanical dither.

cs.RO↗

arXiv2024

Angle-Resolved Magneto-Chiral Anisotropy in a Non-Centrosymmetric Atomic Layer Superlattice

Chirality in solid-state materials has sparked significant interest due to potential applications of topologically-protected chiral states in next-generation information technology. The electrical magneto-chiral effect (eMChE), arising from relativistic spin-orbit interactions, shows great promise for developing chiral materials and devices for electronic integration. Here we demonstrate an angle-resolved eMChE in an A-B-C-C type atomic-layer superlattice lacking time and space inversion symmetry. We observe non-superimposable enantiomers of left-handed and right-handed tilted uniaxial magnetic anisotropy as the sample rotates under static fields, with the tilting angle reaching a striking 45 degree. Magnetic force microscopy and atomistic simulations correlate the tilt to the emergence and evolution of chiral spin textures. The Dzyaloshinskii-Moriya interaction lock effect in competition with Zeeman effect is demonstrated to be responsible for the angle-resolved eMChE. Our findings open up a new horizon for engineering angle-resolved magneto-chiral anisotropy, shedding light on the development of novel angle-resolved sensing or writing techniques in chiral spintronics.

cond-mat.mtrl-sci↗

arXiv2023

The s$^\pm$-Wave Pairing and the Destructive Role of Apical-Oxygen Deficiencies in La$_3$Ni$_2$O$_7$ Under Pressure

Recently, the bilayer perovskite nickelate La$_3$Ni$_2$O$_7$ has been reported to show evidence of high-temperature superconductivity (SC) under a moderate pressure of about 14 GPa. To investigate the superconducting mechanism, pairing symmetry, and the role of apical-oxygen deficiencies in this material, we perform a random-phase-approximation based study on a bilayer model consisting of the $d_{x^2-y^2}$ and $d_{3z^2-r^2}$ orbitals of Ni atoms in both the pristine crystal and the crystal with apical-oxygen deficiencies. Our analysis reveals an $s^{\pm}$-wave pairing symmetry driven by spin fluctuations. The crucial role of pressure lies in that it induces the emergence of the $γ$-pocket, which is involved in the strongest Fermi-surface nesting. We further found the emergence of local moments in the vicinity of apical-oxygen deficiencies, which significantly suppresses the $T_c$. Therefore, it is possible to significantly enhance the $T_c$ by eliminating oxygen deficiencies during the synthesis of the samples.

cond-mat.supr-con↗

arXiv2023

Twofold Symmetry Observed in Bi$_{2}$Te$_{3}$/FeTe Interfacial Superconductor

Superconducting pairing symmetry are crucial in understanding the microscopic superconducting mechanism of a superconductor. Here we report the observation of a twofold superconducting gap symmetry in an interfacial superconductor Bi$_{2}$Te$_{3}$/FeTe, by employing quasiparticle interference (QPI) technique in scanning tunneling microscopy and macroscopic magnetoresistance measurements. The QPI patterns corresponding to energies inside and outside the gap reveal a clear anisotropic superconducting gap. Furthermore, both the in-plane angle-dependent magnetoresistance and in-plane upper critical field exhibit a clear twofold symmetry. This twofold symmetry align with the Te-Te direction in FeTe, which weakens the possible generation by bi-collinear antiferromagnetism order. Our finding provides key information in further understanding of the topological properties in Bi$_{2}$Te$_{3}$/FeTe superconducting system and propels further theoretical interests in the paring mechanism in the system.

cond-mat.supr-con↗

arXiv2023

Left-invariant Hermitian connections on Lie groups with almost Hermitian structures

Left-invariant Hermitian and Gauduchon connections are studied on an arbitrary Lie group $G$ equipped with an arbitrary left-invariant almost Hermitian structure $(\langle\cdot,\cdot\rangle,J)$. The space of left-invariant Hermitian connections is shown to be in one-to-one correspondence with the space $\wedge^{(1,1)}\mathfrak{g}^\ast\otimes \mathfrak{g}$ of left-invariant 2-forms of type (1,1) (with respect to $J$) with values in $\mathfrak{g}:=\mbox{Lie}(G)$. Explicit formulas are obtained for the torsion components of every Hermitian and Gauduchon connection with respect to a convenient choice of left-invariant frame on $G$. The curvature of Gauduchon connections is studied for the special case $G=H\times A$, where $H$ is an arbitrary $n$-dimensional Lie group, $A$ is an arbitrary $n$-dimensional abelian Lie group, and the almost complex structure is totally real with respect to $\mathfrak{h}:=\mbox{Lie}(H)$. When $H$ is compact, it is shown that $H\times A$ admits a left-invariant (strictly) almost Hermitian structure $(\langle\cdot,\cdot\rangle,J)$ such that the Gauduchon connection corresponding to the Strominger (or Bismut) connection in the integrable case is precisely the trivial left-invariant connection and, in addition, has totally skew-symmetric torsion. The almost Hermitian structure $(\langle\cdot,\cdot\rangle,J)$ on $H\times A$ is shown to satisfy the \textit{strong Kähler with torsion} condition. Furthermore, the affine line of Gauduchon connections on $H\times A$ with the aforementioned almost Hermitian structure is also shown to contain a (nontrivial) flat connection.

math.DG↗