arXiv subjects

Xiangyue Liu

Publications and source records attributed to Xiangyue Liu.

16 recordsLinked to original sources

arXiv2026

Rosetta: Composable Native Multimodal Pretraining

Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge. However, accommodating continuous generative objectives alongside discrete understanding tasks causes severe gradient conflicts. Existing architectures, including standard Mixture-of-Experts (MoE), are highly susceptible to representation overwriting. Even structurally partitioned paradigms like Mixture-of-Transformers (MoT) remain vulnerable to catastrophic forgetting, severely impeding multimodal scalability. In this work, we introduce Rosetta, a composable native multimodal pretraining framework designed for seamless and non-destructive modality expansion. Rosetta adopts a modular paradigm where core foundational knowledge is preserved within global shared experts, while modality-specific capabilities are distributed across plug-and-play experts. To guarantee non-destructive composition, we propose Momentum-Anchored Orthogonal Projection (MAOP). MAOP leverages the optimizer's momentum state as an implicit semantic anchor, selectively neutralizing conflicting gradient components from new modalities while preserving synergistic updates. Extensive evaluations demonstrate that, while standard MoE and MoT architectures suffer catastrophic forgetting of previously acquired knowledge, Rosetta robustly preserves established language and visual understanding. Furthermore, it delivers superior image generation and unlocks cross-modal synergy, paving the way for truly composable and unified multimodal foundation models. To facilitate further multimodal research, we release our code and checkpoints to the community. Project page at https://rosetta-lmm.github.io/.

cs.CV↗

arXiv2026

Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts. While existing paradigms like Mixture-of-Transformers (MoT) mitigate this conflict through structural isolation, they fundamentally sever cross-modal synergy and suffer from capacity fragmentation. In this work, we present Symbiotic-MoE, a unified pre-training framework that resolves task interference within a native multimodal Mixture-of-Experts (MoE) Transformers architecture with zero-parameter overhead. We first identify that standard MoE tuning leads to routing collapse, where generative gradients dominate expert utilization. To address this, we introduce Modality-Aware Expert Disentanglement, which partitions experts into task-specific groups while utilizing shared experts as a multimodal semantic bridge. Crucially, this design allows shared experts to absorb fine-grained visual semantics from generative tasks to enrich textual representations. To optimize this, we propose a Progressive Training Strategy featuring differential learning rates and early-stage gradient shielding. This mechanism not only shields pre-trained knowledge from early volatility but eventually transforms generative signals into constructive feedback for understanding. Extensive experiments demonstrate that Symbiotic-MoE achieves rapid generative convergence while unlocking cross-modal synergy, boosting inherent understanding with remarkable gains on MMLU and OCRBench.

cs.CV↗

arXiv2026

Continuous thermochemical sources of AlF molecules

The AlF molecule, currently subject to laser cooling and trapping efforts, has the advantage that it can be efficiently produced in a thermochemical reaction between sublimated aluminum trifluoride and aluminum metal. Here we present a series of experiments with continuous molecular beam sources of AlF, utilising this reaction. We demonstrate a compact AlF molecular beam oven whose total far-field brightness is $5\times 10^{15}$ molecules per steradian per second at 923~K, just below the melting temperature of aluminum metal. The continuous output from the oven begins to exceed the peak brightness of a jet-cooled, ablation-based supersonic AlF source for the $v=0$, $J=7$ level, and we obtain an excellent signal-to-noise ratio with the oven in pulsed laser ionisation spectroscopy experiments. By delivering flux from the oven into a cryogenic Ne buffer gas cell, we lower the rotational temperature of the beam to around 30~K and reduce its most probable forward velocity from 600~ms$^{-1}$ to 200~ms$^{-1}$. In addition, we demonstrate that AlF can be made in a simple dispenser package, and observe that molecules thermalise to the laboratory temperature after colliding with vacuum chamber walls of the experiment. The resulting transient AlF vapour may enable direct loading of a molecular magneto-optical trap.

physics.atom-ph↗

arXiv2025

NoiseAR: AutoRegressing Initial Noise Prior for Diffusion Models

Diffusion models have emerged as powerful generative frameworks, creating data samples by progressively denoising an initial random state. Traditionally, this initial state is sampled from a simple, fixed distribution like isotropic Gaussian, inherently lacking structure and a direct mechanism for external control. While recent efforts have explored ways to introduce controllability into the diffusion process, particularly at the initialization stage, they often rely on deterministic or heuristic approaches. These methods can be suboptimal, lack expressiveness, and are difficult to scale or integrate into more sophisticated optimization frameworks. In this paper, we introduce NoiseAR, a novel method for AutoRegressive Initial Noise Prior for Diffusion Models. Instead of a static, unstructured source, NoiseAR learns to generate a dynamic and controllable prior distribution for the initial noise. We formulate the generation of the initial noise prior's parameters as an autoregressive probabilistic modeling task over spatial patches or tokens. This approach enables NoiseAR to capture complex spatial dependencies and introduce learned structure into the initial state. Crucially, NoiseAR is designed to be conditional, allowing text prompts to directly influence the learned prior, thereby achieving fine-grained control over the diffusion initialization. Our experiments demonstrate that NoiseAR can generate initial noise priors that lead to improved sample quality and enhanced consistency with conditional inputs, offering a powerful, learned alternative to traditional random initialization. A key advantage of NoiseAR is its probabilistic formulation, which naturally supports seamless integration into probabilistic frameworks like Markov Decision Processes and Reinforcement Learning. Our code will be available at https://github.com/HKUST-SAIL/NoiseAR/

cs.LG↗

arXiv2025

GaussianAvatar-Editor: Photorealistic Animatable Gaussian Head Avatar Editor

We introduce GaussianAvatar-Editor, an innovative framework for text-driven editing of animatable Gaussian head avatars that can be fully controlled in expression, pose, and viewpoint. Unlike static 3D Gaussian editing, editing animatable 4D Gaussian avatars presents challenges related to motion occlusion and spatial-temporal inconsistency. To address these issues, we propose the Weighted Alpha Blending Equation (WABE). This function enhances the blending weight of visible Gaussians while suppressing the influence on non-visible Gaussians, effectively handling motion occlusion during editing. Furthermore, to improve editing quality and ensure 4D consistency, we incorporate conditional adversarial learning into the editing process. This strategy helps to refine the edited results and maintain consistency throughout the animation. By integrating these methods, our GaussianAvatar-Editor achieves photorealistic and consistent results in animatable 4D Gaussian editing. We conduct comprehensive experiments across various subjects to validate the effectiveness of our proposed techniques, which demonstrates the superiority of our approach over existing methods. More results and code are available at: [Project Link](https://xiangyueliu.github.io/GaussianAvatar-Editor/).

cs.CV↗

arXiv2024

GenN2N: Generative NeRF2NeRF Translation

We present GenN2N, a unified NeRF-to-NeRF translation framework for various NeRF translation tasks such as text-driven NeRF editing, colorization, super-resolution, inpainting, etc. Unlike previous methods designed for individual translation tasks with task-specific schemes, GenN2N achieves all these NeRF editing tasks by employing a plug-and-play image-to-image translator to perform editing in the 2D domain and lifting 2D edits into the 3D NeRF space. Since the 3D consistency of 2D edits may not be assured, we propose to model the distribution of the underlying 3D edits through a generative model that can cover all possible edited NeRFs. To model the distribution of 3D edited NeRFs from 2D edited images, we carefully design a VAE-GAN that encodes images while decoding NeRFs. The latent space is trained to align with a Gaussian distribution and the NeRFs are supervised through an adversarial loss on its renderings. To ensure the latent code does not depend on 2D viewpoints but truly reflects the 3D edits, we also regularize the latent code through a contrastive learning scheme. Extensive experiments on various editing tasks show GenN2N, as a universal framework, performs as well or better than task-specific specialists while possessing flexible generative power. More results on our project page: https://xiangyueliu.github.io/GenN2N/

cs.CV↗

arXiv2023

KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view Stereo

Supervised multi-view stereo (MVS) methods have achieved remarkable progress in terms of reconstruction quality, but suffer from the challenge of collecting large-scale ground-truth depth. In this paper, we propose a novel self-supervised training pipeline for MVS based on knowledge distillation, termed KD-MVS, which mainly consists of self-supervised teacher training and distillation-based student training. Specifically, the teacher model is trained in a self-supervised fashion using both photometric and featuremetric consistency. Then we distill the knowledge of the teacher model to the student model through probabilistic knowledge transferring. With the supervision of validated knowledge, the student model is able to outperform its teacher by a large margin. Extensive experiments performed on multiple datasets show our method can even outperform supervised methods.

cs.CV↗

arXiv2023

Molecular dynamics-driven global tetra-atomic potential energy surfaces: Application to the AlF dimer

In this work, we present a general machine learning approach for full-dimensional potential energy surfaces for tetra-atomic systems. Our method employs an active learning scheme trained on {\it ab initio} points, which size grows based on the accuracy required. The training points are selected based on molecular dynamics simulations, choosing the most suitable configurations for different collision energy and mapping the most relevant part of the potential energy landscape of the system. The present approach does not require long-range information and is entirely general. As an example, we provide the full-dimensional AlF-AlF potential energy surface, requiring $\lesssim 0.1\%$ of the configurations to be calculated {\it ab initio}. Furthermore, we analyze the general properties of the AlF-AlF system, finding key difference with other reported results on CaF or bi-alkali dimers.

physics.chem-ph↗

arXiv2022

Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives

Recently, Implicit Neural Representations (INRs) parameterized by neural networks have emerged as a powerful and promising tool to represent different kinds of signals due to its continuous, differentiable properties, showing superiorities to classical discretized representations. However, the training of neural networks for INRs only utilizes input-output pairs, and the derivatives of the target output with respect to the input, which can be accessed in some cases, are usually ignored. In this paper, we propose a training paradigm for INRs whose target output is image pixels, to encode image derivatives in addition to image values in the neural network. Specifically, we use finite differences to approximate image derivatives. We show how the training paradigm can be leveraged to solve typical INRs problems, i.e., image regression and inverse rendering, and demonstrate this training paradigm can improve the data-efficiency and generalization capabilities of INRs. The code of our method is available at \url{https://github.com/megvii-research/Sobolev_INRs}.

cs.CV↗

arXiv2022

Spectroscopic characterization of the a$^3Π$ state of aluminum monofluoride

Spectroscopic studies of aluminum monofluoride (AlF) have revealed its highly favorable properties for direct laser cooling. All $Q$ lines of the strong A$^1Π$ $\leftarrow$ X$^1Σ^+$ transition around 227~nm are rotationally closed and thereby suitable for the main cooling cycle. The same holds for the narrow, spin-forbidden a$^3Π$ $\leftarrow$ X$^1Σ^+$ transition around 367 nm which has a recoil limit in the micro Kelvin range. We here report on the spectroscopic characterization of the lowest rotational levels in the a$^3Π$ state of AlF for $v=0-8$ using a jet-cooled, pulsed molecular beam. An accidental AC Stark shift is observed on the a$^3Π_0, v=4$ $\leftarrow$ X$^1Σ^+, v=4$ band. By using time-delayed ionization for state-selective detection of the molecules in the metastable a$^3Π$ state at different points along the molecular beam, the radiative lifetime of the a$^3Π_1, v=0, J=1$ level is experimentally determined as $τ=1.89 \pm 0.15$~ms. A laser/radio-frequency multiple resonance ionization scheme is employed to determine the hyperfine splittings in the a$^3Π_1, v=5$ level. The experimentally derived hyperfine parameters are compared to the outcome of quantum chemistry calculations. A spectral line with a width of 1.27 kHz is recorded between hyperfine levels in the a$^3Π, v=0$ state. These measurements benchmark the electronic potential of the a$^3Π$ state and yield accurate values for the photon scattering rate and for the elements of the Franck-Condon matrix of the a$^3Π$ $-$ X$^1Σ^+$ system.

physics.chem-ph↗

arXiv2021

TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformers

In this paper, we present TransMVSNet, based on our exploration of feature matching in multi-view stereo (MVS). We analogize MVS back to its nature of a feature matching task and therefore propose a powerful Feature Matching Transformer (FMT) to leverage intra- (self-) and inter- (cross-) attention to aggregate long-range context information within and across images. To facilitate a better adaptation of the FMT, we leverage an Adaptive Receptive Field (ARF) module to ensure a smooth transit in scopes of features and bridge different stages with a feature pathway to pass transformed features and gradients across different scales. In addition, we apply pair-wise feature correlation to measure similarity between features, and adopt ambiguity-reducing focal loss to strengthen the supervision. To the best of our knowledge, TransMVSNet is the first attempt to leverage Transformer into the task of MVS. As a result, our method achieves state-of-the-art performance on DTU dataset, Tanks and Temples benchmark, and BlendedMVS dataset. The code of our method will be made available at https://github.com/MegviiRobot/TransMVSNet .

cs.CV↗

arXiv2021

Is CCSD(T) a proper standard for dipole moment calculations? An analysis considering diverse diatomic species

Coupled cluster with single, double, and perturbative triple excitations [CCSD(T)] has been extensively employed as the reference method in benchmarking different quantum chemistry methods. In this work, we test the accuracy of CCSD(T) calculating ground state electric dipole moments at the extrapolated complete basis set (CBS) limit. The calculated dipole moments have been compared to an experimental dataset consisted of diatomic molecules with various kinds of bond natures and spin configurations. As a result, to reach a satisfactory agreement with experimental dipole moments, core-correlations should be included for some molecules. However, even when core-correlations are included, the predicted dipole moment deviates considerably from the experimental values for molecules involving transition metal atoms.

physics.chem-ph↗

arXiv2020

2nd Place Solution to Instance Segmentation of IJCAI 3D AI Challenge 2020

Compared with MS-COCO, the dataset for the competition has a larger proportion of large objects which area is greater than 96x96 pixels. As getting fine boundaries is vitally important for large object segmentation, Mask R-CNN with PointRend is selected as the base segmentation framework to output high-quality object boundaries. Besides, a better engine that integrates ResNeSt, FPN and DCNv2, and a range of effective tricks that including multi-scale training and test time augmentation are applied to improve segmentation performance. Our best performance is an ensemble of four models (three PointRend-based models and SOLOv2), which won the 2nd place in IJCAI-PRICAI 3D AI Challenge 2020: Instance Segmentation.

cs.CV↗

arXiv2020

A data-driven approach to determine dipole moments of diatomic molecules

We present a data-driven approach for the prediction of the electric dipole moment of diatomic molecules, which is one of the most relevant molecular properties. In particular, we apply Gaussian process regression to a novel dataset to show that dipole moments of diatomic molecules can be learned, and hence predicted, with a relative error <5%. The dataset contains the dipole moment of 162 diatomic molecules, the most exhaustive and unbiased dataset of dipole moments up to date. Our findings show that the dipole moment of diatomic molecules depends on atomic properties of the constituents atoms: electron affinity and ionization potential, as well as on (a feature related to) the first derivative of the electronic kinetic energy at the equilibrium distance.

physics.chem-ph↗

arXiv2020

On the universality of spectroscopic constants of diatomic molecules

We show, through a machine learning approach, that the equilibrium distance, harmonic vibrational frequency, and binding energy of diatomic molecules are universally related. In particular, the relationships between spectroscopic constants are valid independently of the molecular bond. However, they depend strongly on the group and period of the constituent atoms. As a result, we show that by employing the group and period of atoms within a molecule, the spectroscopic constants are predicted with an accuracy of $\lesssim 5\%$. Finally, the same universal relationships are satisfied when spectroscopic constants from {\it ab initio} and density functional theory (DFT) electronic structure methods are employed.

physics.chem-ph↗

arXiv2018

NOMAD 2018 Kaggle Competition: Solving Materials Science Challenges Through Crowd Sourcing

Machine learning (ML) is increasingly used in the field of materials science, where statistical estimates of computed properties are employed to rapidly examine the chemical space for new compounds. However, a systematic comparison of several ML models for this domain has been hindered by the scarcity of appropriate datasets of materials properties, as well as the lack of thorough benchmarking studies. To address this, a public data-analytics competition was organized by the Novel Materials Discovery (NOMAD) Centre of Excellence and hosted by the on-line platform Kaggle using a dataset of $3\,000$ (Al$_x$ Ga$_y$ In$_z$)$_2$ O$_3$ compounds (with $x+y+z = 1$). The aim of this challenge was to identify the best ML model for the prediction of two key physical properties that are relevant for optoelectronic applications: the electronic band gap energy and the crystalline formation energy. In this contribution, we present a summary of the top three ML approaches of the competition including the 1st place solution based on a crystal graph representation that is new for ML of the properties of materials. The 2nd place model combined many candidate descriptors from a set of compositional, atomic environment-based, and average structural properties with the light gradient-boosting machine regression model. The 3rd place model employed the smooth overlap of atomic positions representation with a neural network. To gain insight into whether the representation or the regression model determines the overall model performance, nine ML models obtained by combining the representations and regression models of the top three approaches were compared by looking at the correlations among prediction errors. At fixed representation, the largest correlation is observed in predictions made with kernel ridge regression and neural network, reflecting a similar performance on the same test set samples.

cond-mat.mtrl-sci↗