arXiv subjects

Weinan E

Publications and source records attributed to Weinan E.

At least 55 records · Page 3Linked to original sources

arXiv2025

Scalable Complexity Control Facilitates Reasoning Ability of LLMs

The reasoning ability of large language models (LLMs) has been rapidly advancing in recent years, attracting interest in more fundamental approaches that can reliably enhance their generalizability. This work demonstrates that model complexity control, conveniently implementable by adjusting the initialization rate and weight decay coefficient, improves the scaling law of LLMs consistently over varying model sizes and data sizes. This gain is further illustrated by comparing the benchmark performance of 2.4B models pretrained on 1T tokens with different complexity hyperparameters. Instead of fixing the initialization std, we found that a constant initialization rate (the exponent of std) enables the scaling law to descend faster in both model and data sizes. These results indicate that complexity control is a promising direction for the continual advancement of LLMs.

cs.LG↗

arXiv2025

PaSa: An LLM Agent for Comprehensive Academic Paper Search

We introduce PaSa, an advanced Paper Search agent powered by large language models. PaSa can autonomously make a series of decisions, including invoking search tools, reading papers, and selecting relevant references, to ultimately obtain comprehensive and accurate results for complex scholar queries. We optimize PaSa using reinforcement learning with a synthetic dataset, AutoScholarQuery, which includes 35k fine-grained academic queries and corresponding papers sourced from top-tier AI conference publications. Additionally, we develop RealScholarQuery, a benchmark collecting real-world academic queries to assess PaSa performance in more realistic scenarios. Despite being trained on synthetic data, PaSa significantly outperforms existing baselines on RealScholarQuery, including Google, Google Scholar, Google with GPT-4o for paraphrased queries, ChatGPT (search-enabled GPT-4o), GPT-o1, and PaSa-GPT-4o (PaSa implemented by prompting GPT-4o). Notably, PaSa-7B surpasses the best Google-based baseline, Google with GPT-4o, by 37.78% in recall@20 and 39.90% in recall@50, and exceeds PaSa-GPT-4o by 30.36% in recall and 4.25% in precision. Model, datasets, and code are available at https://github.com/bytedance/pasa.

cs.IR↗

arXiv2025

RARE: Retrieval-Augmented Reasoning Modeling

Domain-specific intelligence demands specialized knowledge and sophisticated reasoning for problem-solving, posing significant challenges for large language models (LLMs) that struggle with knowledge hallucination and inadequate reasoning capabilities under constrained parameter budgets. Inspired by Bloom's Taxonomy in educational theory, we propose Retrieval-Augmented Reasoning Modeling (RARE), a novel paradigm that decouples knowledge storage from reasoning optimization. RARE externalizes domain knowledge to retrievable sources and internalizes domain-specific reasoning patterns during training. Specifically, by injecting retrieved knowledge into training prompts with masked losses, RARE transforms learning objectives from rote memorization to contextualized reasoning. It enables models to bypass parameter-intensive memorization and prioritize the development of higher-order cognitive processes. Extensive experiments demonstrate that lightweight RARE-trained models (e.g., Llama-3.1-8B) could achieve state-of-the-art performance, surpassing retrieval-augmented GPT-4 and DeepSeek-R1 up to approximately 20\% accuracy. RARE establishes a paradigm shift where maintainable external knowledge bases synergize with compact, reasoning-optimized models, collectively driving more scalable domain-specific intelligence.

cs.CL↗

arXiv2025

A brief review of the Deep BSDE method for solving high-dimensional partial differential equations

High-dimensional partial differential equations (PDEs) pose significant challenges for numerical computation due to the curse of dimensionality, which limits the applicability of traditional mesh-based methods. Since 2017, the Deep BSDE method has introduced deep learning techniques that enable the effective solution of nonlinear PDEs in very high dimensions. This innovation has sparked considerable interest in using neural networks for high-dimensional PDEs, making it an active area of research. In this short review, we briefly sketch the Deep BSDE method, its subsequent developments, and future directions for the field.

math.NA↗

arXiv2025

Thermal disorder and phonon softening in the ferroelectric phase transition of lead titanate

We report a molecular dynamics study of ab initio quality of the ferroelectric phase transition in crystalline PbTiO3. We model anharmonicity accurately in terms of potential energy and polarization surfaces trained on density functional theory data with modern machine learning techniques. Our simulations demonstrate that the transition has a strong order-disorder character, in agreement with diffraction experiments, and provide fresh insight into the approach to equilibrium across the phase transition. We find that the emergence and disappearance of the macroscopic polarization is driven by dipolar switching at the nanometer scale. We also computed the infrared optical absorption spectra in both the ferroelectric and the paraelectric phases, finding good agreement with the experimental Raman frequencies. Often, the almost ideal displacive character of the soft mode detected by Raman scattering in the paraelectric phase has been contrasted with the order-disorder character of the transition suggested by diffraction experiments. We settle this issue by showing that the soft mode coexists with a strong Debye relaxation associated with thermal disordering of the dipoles. The Debye relaxation feature is centered at zero frequency and appears near the transition temperature in both the ferroelectric and the paraelectric phases.

cond-mat.mtrl-sci↗

arXiv2025

Discovery of High-Temperature Superconducting Ternary Hydrides via Deep Learning

The discovery of novel high-temperature superconductor materials holds transformative potential for a wide array of technological applications. However, the combinatorially vast chemical and configurational search space poses a significant bottleneck for both experimental and theoretical investigations. In this study, we employ the design of high-temperature ternary superhydride superconductors as a representative case to demonstrate how this challenge can be well addressed through a deep-learning-driven theoretical framework. This framework integrates high-throughput crystal structure exploration, physics-informed screening, and accurate prediction of superconducting critical temperatures. Our approach enabled the exploration of approximately 36 million ternary hydride structures across a chemical space of 29 elements, leading to the identification of 144 potential high-Tc superconductors with predicted Tc > 200 K and superior thermodynamic stability at 200 GPa. Among these, 129 compounds spanning 27 novel structural prototypes are reported for the first time, representing a significant expansion of the known structural landscape for hydride superconductors. This work not only greatly expands the known repertoire of high-Tc hydride superconductors but also establishes a scalable and efficient methodology for navigating the complex landscape of multinary hydrides.

cond-mat.supr-con↗

arXiv2025

Strategic priorities for transformative progress in advancing biology with proteomics and artificial intelligence

Artificial intelligence (AI) is transforming scientific research, including proteomics. Advances in mass spectrometry (MS)-based proteomics data quality, diversity, and scale, combined with groundbreaking AI techniques, are unlocking new challenges and opportunities in biological discovery. Here, we highlight key areas where AI is driving innovation, from data analysis to new biological insights. These include developing an AI-friendly ecosystem for proteomics data generation, sharing, and analysis; improving peptide and protein identification and quantification; characterizing protein-protein interactions and protein complexes; advancing spatial and perturbation proteomics; integrating multi-omics data; and ultimately enabling AI-empowered virtual cells.

q-bio.OT↗

arXiv2025

How Transformers Get Rich: Approximation and Dynamics Analysis

Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remains limited. A recent work (Elhage et al., 2021) identified a ``rich'' in-context mechanism known as induction head, contrasting with ``lazy'' $n$-gram models that overlook long-range dependencies. In this work, we provide both approximation and dynamics analyses of how transformers implement induction heads. In the {\em approximation} analysis, we formalize both standard and generalized induction head mechanisms, and examine how transformers can efficiently implement them, with an emphasis on the distinct role of each transformer submodule. For the {\em dynamics} analysis, we study the training dynamics on a synthetic mixed target, composed of a 4-gram and an in-context 2-gram component. This controlled setting allows us to precisely characterize the entire training process and uncover an {\em abrupt transition} from lazy (4-gram) to rich (induction head) mechanisms as training progresses.

cs.LG↗

arXiv2025

Solving Optimal Control Problems of Rigid-Body Dynamics with Collisions Using the Hybrid Minimum Principle

Collisions are common in many dynamical systems with real applications. They can be formulated as hybrid dynamical systems with discontinuities automatically triggered when states transverse certain manifolds. We present an algorithm for the optimal control problem of such hybrid dynamical systems based on solving the equations derived from the hybrid minimum principle (HMP). The algorithm is an iterative scheme following the spirit of the method of successive approximations (MSA), and it is robust to undesired collisions observed in the initial guesses. We propose several techniques to address the additional numerical challenges introduced by the presence of discontinuities. The algorithm is tested on disc collision problems whose optimal solutions exhibit one or multiple collisions. Linear convergence in terms of iteration steps and asymptotic first-order accuracy in terms of time discretization are observed when the algorithm is implemented with the forward-Euler scheme. The numerical results demonstrate that the proposed algorithm has better accuracy and convergence than direct methods based on gradient descent. Furthermore, the algorithm is also simpler, more accurate, and more stable than a deep reinforcement learning method.

math.OC↗

arXiv2025

Intelligent System for Automated Molecular Patent Infringement Assessment

Automated drug discovery offers significant potential for accelerating the development of novel therapeutics by substituting labor-intensive human workflows with machine-driven processes. However, molecules generated by artificial intelligence may unintentionally infringe on existing patents, posing legal and financial risks that impede the full automation of drug discovery pipelines. This paper introduces PatentFinder, a novel multi-agent and tool-enhanced intelligence system that can accurately and comprehensively evaluate small molecules for patent infringement. PatentFinder features five specialized agents that collaboratively analyze patent claims and molecular structures with heuristic and model-based tools, generating interpretable infringement reports. To support systematic evaluation, we curate MolPatent-240, a benchmark dataset tailored for patent infringement assessment algorithms. On this benchmark, PatentFinder outperforms baseline methods that rely solely on large language models or specialized chemical tools, achieving a 13.8% improvement in F1-score and a 12% increase in accuracy. Additionally, PatentFinder autonomously generates detailed and interpretable patent infringement reports, showcasing enhanced accuracy and improved interpretability. The high accuracy and interpretability of PatentFinder make it a valuable and reliable tool for automating patent infringement assessments, offering a practical solution for integrating patent protection analysis into the drug discovery pipeline.

cs.LG↗

arXiv2024

Deep learning tight-binding approach for large-scale electronic simulations at finite temperatures with $ab$ $initio$ accuracy

Simulating electronic behavior in materials and devices with realistic large system sizes remains a formidable task within the $ab$ $initio$ framework due to its computational intensity. Here we show DeePTB, an efficient deep learning-based tight-binding approach with $ab$ $initio$ accuracy to address this issue. By training on structural data and corresponding $ab$ $initio$ eigenvalues, the DeePTB model can efficiently predict tight-binding Hamiltonians for unseen structures, enabling efficient simulations of large-size systems under external perturbations such as finite temperatures and strain. This capability is vital for semiconductor band gap engineering and materials design. When combined with molecular dynamics, DeePTB facilitates efficient and accurate finite-temperature simulations of both atomic and electronic behavior simultaneously. This is demonstrated by computing the temperature-dependent electronic properties of a gallium phosphide system with $10^6$ atoms. The availability of DeePTB bridges the gap between accuracy and scalability in electronic simulations, potentially advancing materials science and related fields by enabling large-scale electronic structure calculations.

cond-mat.mtrl-sci↗

arXiv2024

Improving Generalization and Convergence by Enhancing Implicit Regularization

In this work, we propose an Implicit Regularization Enhancement (IRE) framework to accelerate the discovery of flat solutions in deep learning, thereby improving generalization and convergence. Specifically, IRE decouples the dynamics of flat and sharp directions, which boosts the sharpness reduction along flat directions while maintaining the training stability in sharp directions. We show that IRE can be practically incorporated with {\em generic base optimizers} without introducing significant computational overload. Experiments show that IRE consistently improves the generalization performance for image classification tasks across a variety of benchmark datasets (CIFAR-10/100, ImageNet) and models (ResNets and ViTs). Surprisingly, IRE also achieves a $2\times$ {\em speed-up} compared to AdamW in the pre-training of Llama models (of sizes ranging from 60M to 229M) on datasets including Wikitext-103, Minipile, and Openwebtext. Moreover, we provide theoretical guarantees, showing that IRE can substantially accelerate the convergence towards flat minima in Sharpness-aware Minimization (SAM).

cs.LG↗

arXiv2024

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads. These theoretical insights are validated experimentally and offer natural suggestions for alternative architectures.

cs.LG↗

arXiv2024

On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis

We study the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data. We consider the simple but representative setting of using continuous-time linear RNNs to learn from data generated by linear relationships. Mathematically, the latter can be understood as a sequence of linear functionals. We prove a universal approximation theorem of such linear functionals, and characterize the approximation rate and its relation with memory. Moreover, we perform a fine-grained dynamical analysis of training linear RNNs, which further reveal the intricate interactions between memory and learning. A unifying theme uncovered is the non-trivial effect of memory, a notion that can be made precise in our framework, on approximation and optimization: when there is long term memory in the target, it takes a large number of neurons to approximate it. Moreover, the training process will suffer from slow downs. In particular, both of these effects become exponentially more pronounced with memory - a phenomenon we call the "curse of memory". These analyses represent a basic step towards a concrete mathematical understanding of new phenomenon that may arise in learning temporal relationships using recurrent architectures.

cs.LG↗

arXiv2024

Towards a Unified Benchmark and Framework for Deep Learning-Based Prediction of Nuclear Magnetic Resonance Chemical Shifts

The study of structure-spectrum relationships is essential for spectral interpretation, impacting structural elucidation and material design. Predicting spectra from molecular structures is challenging due to their complex relationships. Herein, we introduce NMRNet, a deep learning framework using the SE(3) Transformer for atomic environment modeling, following a pre-training and fine-tuning paradigm. To support the evaluation of NMR chemical shift prediction models, we have established a comprehensive benchmark based on previous research and databases, covering diverse chemical systems. Applying NMRNet to these benchmark datasets, we achieve state-of-the-art performance in both liquid-state and solid-state NMR datasets, demonstrating its robustness and practical utility in real-world scenarios. This marks the first integration of solid and liquid state NMR within a unified model architecture, highlighting the need for domainspecific handling of different atomic environments. Our work sets a new standard for NMR prediction, advancing deep learning applications in analytical and structural chemistry.

physics.comp-ph↗

arXiv2024

DPA-2: a large atomic model as a multi-task learner

The rapid advancements in artificial intelligence (AI) are catalyzing transformative changes in atomic modeling, simulation, and design. AI-driven potential energy models have demonstrated the capability to conduct large-scale, long-duration simulations with the accuracy of ab initio electronic structure methods. However, the model generation process remains a bottleneck for large-scale applications. We propose a shift towards a model-centric ecosystem, wherein a large atomic model (LAM), pre-trained across multiple disciplines, can be efficiently fine-tuned and distilled for various downstream tasks, thereby establishing a new framework for molecular modeling. In this study, we introduce the DPA-2 architecture as a prototype for LAMs. Pre-trained on a diverse array of chemical and materials systems using a multi-task approach, DPA-2 demonstrates superior generalization capabilities across multiple downstream tasks compared to the traditional single-task pre-training and fine-tuning methodologies. Our approach sets the stage for the development and broad application of LAMs in molecular and materials simulation research.

physics.chem-ph↗

arXiv2024

Invertible Coarse Graining with Physics-Informed Generative Artificial Intelligence

Multiscale molecular modeling is widely applied in scientific research of molecular properties over large time and length scales. Two specific challenges are commonly present in multiscale modeling, provided that information between the coarse and fine representations of molecules needs to be properly exchanged: One is to construct coarse grained models by passing information from the fine to coarse levels; the other is to restore finer molecular details given coarse grained configurations. Although these two problems are commonly addressed independently, in this work, we present a theory connecting them, and develop a methodology called Cycle Coarse Graining (CCG) to solve both problems in a unified manner. In CCG, reconstruction can be achieved via a tractable deep generative model, allowing retrieval of fine details from coarse-grained simulations. The reconstruction in turn delivers better coarse-grained models which are informed of the fine-grained physics, and enables calculation of the free energies in a rare-event-free manner. CCG thus provides a systematic way for multiscale molecular modeling, where the finer details of coarse-grained simulations can be efficiently retrieved, and the coarse-grained models can be improved consistently.

physics.comp-ph↗

arXiv2024

Uni-ELF: A Multi-Level Representation Learning Framework for Electrolyte Formulation Design

Advancements in lithium battery technology heavily rely on the design and engineering of electrolytes. However, current schemes for molecular design and recipe optimization of electrolytes lack an effective computational-experimental closed loop and often fall short in accurately predicting diverse electrolyte formulation properties. In this work, we introduce Uni-ELF, a novel multi-level representation learning framework to advance electrolyte design. Our approach involves two-stage pretraining: reconstructing three-dimensional molecular structures at the molecular level using the Uni-Mol model, and predicting statistical structural properties (e.g., radial distribution functions) from molecular dynamics simulations at the mixture level. Through this comprehensive pretraining, Uni-ELF is able to capture intricate molecular and mixture-level information, which significantly enhances its predictive capability. As a result, Uni-ELF substantially outperforms state-of-the-art methods in predicting both molecular properties (e.g., melting point, boiling point, synthesizability) and formulation properties (e.g., conductivity, Coulombic efficiency). Moreover, Uni-ELF can be seamlessly integrated into an automatic experimental design workflow. We believe this innovative framework will pave the way for automated AI-based electrolyte design and engineering.

physics.chem-ph↗