arXiv subjects

Xinhao Li

Publications and source records attributed to Xinhao Li.

49 records · Page 3Linked to original sources

arXiv2024

InternVideo2: Scaling Foundation Models for Multimodal Video Understanding

We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue. Our core design is a progressive training approach that unifies the masked video modeling, crossmodal contrastive learning, and next token prediction, scaling up the video encoder size to 6B parameters. At the data level, we prioritize spatiotemporal consistency by semantically segmenting videos and generating video-audio-speech captions. This improves the alignment between video and text. Through extensive experiments, we validate our designs and demonstrate superior performance on over 60 video and audio tasks. Notably, our model outperforms others on various video-related dialogue and long video understanding benchmarks, highlighting its ability to reason and comprehend longer contexts. Code and models are available at https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2/.

cs.CV↗

arXiv2024

ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Video

Adapting image models to the video domain has emerged as an efficient paradigm for solving video recognition tasks. Due to the huge number of parameters and effective transferability of image models, performing full fine-tuning is less efficient and even unnecessary. Thus, recent research is shifting its focus toward parameter-efficient image-to-video adaptation. However, these adaptation strategies inevitably introduce extra computational costs to deal with the domain gap and temporal modeling in videos. In this paper, we present a new adaptation paradigm (ZeroI2V) to transfer the image transformers to video recognition tasks (i.e., introduce zero extra cost to the original models during inference). To achieve this goal, we present two core designs. First, to capture the dynamics in videos and reduce the difficulty of image-to-video adaptation, we exploit the flexibility of self-attention and introduce spatial-temporal dual-headed attention (STDHA). This approach efficiently endows the image transformers with temporal modeling capability at zero extra parameters and computation. Second, to handle the domain gap between images and videos, we propose a linear adaption strategy that utilizes lightweight densely placed linear adapters to fully transfer the frozen image models to video recognition. Thanks to the customized linear design, all newly added adapters could be easily merged with the original modules through structural reparameterization after training, enabling zero extra cost during inference. Extensive experiments on representative fully-supervised and few-shot video recognition benchmarks showcase that ZeroI2V can match or even outperform previous state-of-the-art methods while enjoying superior parameter and inference efficiency.

cs.CV↗

arXiv2024

Bayesian uncertainty analysis for underwater 3D reconstruction with neural radiance fields

Neural radiance fields (NeRFs) are a deep learning technique that can generate novel views of 3D scenes using sparse 2D images from different viewing directions and camera poses. As an extension of conventional NeRFs in underwater environment, where light can get absorbed and scattered by water, SeaThru-NeRF was proposed to separate the clean appearance and geometric structure of underwater scene from the effects of the scattering medium. Since the quality of the appearance and structure of underwater scenes is crucial for downstream tasks such as underwater infrastructure inspection, the reliability of the 3D reconstruction model should be considered and evaluated. Nonetheless, owing to the lack of ability to quantify uncertainty in 3D reconstruction of underwater scenes under natural ambient illumination, the practical deployment of NeRFs in unmanned autonomous underwater navigation is limited. To address this issue, we introduce a spatial perturbation field D_omega based on Bayes' rays in SeaThru-NeRF and perform Laplace approximation to obtain a Gaussian distribution N(0,Sigma) of the parameters omega, where the diagonal elements of Sigma correspond to the uncertainty at each spatial location. We also employ a simple thresholding method to remove artifacts from the rendered results of underwater scenes. Numerical experiments are provided to demonstrate the effectiveness of this approach.

cs.CE↗

arXiv2024

VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Model

With the growth of high-quality data and advancement in visual pre-training paradigms, Video Foundation Models (VFMs) have made significant progress recently, demonstrating their remarkable performance on traditional video understanding benchmarks. However, the existing benchmarks (e.g. Kinetics) and their evaluation protocols are often limited by relatively poor diversity, high evaluation costs, and saturated performance metrics. In this paper, we build a comprehensive benchmark suite to address these issues, namely VideoEval. Specifically, we establish the Video Task Adaption Benchmark (VidTAB) and the Video Embedding Benchmark (VidEB) from two perspectives: evaluating the task adaptability of VFMs under few-shot conditions and assessing their representation power by directly applying to downstream tasks. With VideoEval, we conduct a large-scale study on 20 popular open-source vision foundation models. Our study reveals some insightful findings on VFMs: 1) overall, current VFMs exhibit weak generalization across diverse tasks, 2) increasing video data, whether labeled or weakly-labeled video-text pairs, does not necessarily improve task performance, 3) the effectiveness of some pre-training paradigms may not be fully validated in previous benchmarks, and 4) combining different pre-training paradigms can help improve the generalization capabilities. We believe this study serves as an important complement to the current evaluation for VFMs and offers valuable insights for the future research.

cs.CV↗

arXiv2024

MIMOSA: Multi-constraint Molecule Sampling for Molecule Optimization

Molecule optimization is a fundamental task for accelerating drug discovery, with the goal of generating new valid molecules that maximize multiple drug properties while maintaining similarity to the input molecule. Existing generative models and reinforcement learning approaches made initial success, but still face difficulties in simultaneously optimizing multiple drug properties. To address such challenges, we propose the MultI-constraint MOlecule SAmpling (MIMOSA) approach, a sampling framework to use input molecule as an initial guess and sample molecules from the target distribution. MIMOSA first pretrains two property agnostic graph neural networks (GNNs) for molecule topology and substructure-type prediction, where a substructure can be either atom or single ring. For each iteration, MIMOSA uses the GNNs' prediction and employs three basic substructure operations (add, replace, delete) to generate new molecules and associated weights. The weights can encode multiple constraints including similarity and drug property constraints, upon which we select promising molecules for next iteration. MIMOSA enables flexible encoding of multiple property- and similarity-constraints and can efficiently generate new molecules that satisfy various property constraints and achieved up to 49.6% relative improvement over the best baseline in terms of success rate. The code repository (including readme file, data preprocessing and model construction, evaluation) is available https://github.com/futianfan/MIMOSA.

cs.LG↗

arXiv2024

VideoMamba: State Space Model for Efficient Video Understanding

Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain. The proposed VideoMamba overcomes the limitations of existing 3D convolution neural networks and video transformers. Its linear-complexity operator enables efficient long-term modeling, which is crucial for high-resolution long video understanding. Extensive evaluations reveal VideoMamba's four core abilities: (1) Scalability in the visual domain without extensive dataset pretraining, thanks to a novel self-distillation technique; (2) Sensitivity for recognizing short-term actions even with fine-grained motion differences; (3) Superiority in long-term video understanding, showcasing significant advancements over traditional feature-based models; and (4) Compatibility with other modalities, demonstrating robustness in multi-modal contexts. Through these distinct advantages, VideoMamba sets a new benchmark for video understanding, offering a scalable and efficient solution for comprehensive video understanding. All the code and models are available at https://github.com/OpenGVLab/VideoMamba.

cs.CV↗

arXiv2024

Understanding how off-stoichiometry promotes cation mixing in LiNiO$_2$

Although LiNiO$_2$ is chemically similar to LiCoO$_2$ and offers a nearly identical theoretical capacity, LiNiO$_2$ and related Co-free Ni-rich cathode materials suffer from degradation during electrochemical cycling that has prevented practical use in Li-ion batteries. The observed capacity decay of LiNiO$_2$ has been attributed to the formation of structural defects via Li/Ni cation mixing that reduces cyclability and leads to poor capacity retention. Herein, we investigate the kinetics and thermodynamics of Li/Ni mixing in ideal LiNiO$_2$ and off-stoichiometric Li$_{1-z}$Ni$_{1+z}$O$_2$. We find that ideal LiNiO$_2$ is stable against cation mixing with similar characteristics as LiCoO$_2$. Li/Ni mixing is promoted by extra Ni in the Li layers that cannot be avoided in synthesis. Our study elucidates the crucial role of extra Ni atoms on Li sites in the cation mixing mechanism, an insight that can inform the development of Co-free cathode materials.

cond-mat.mtrl-sci↗

arXiv2024

Learning to (Learn at Test Time)

We reformulate the problem of supervised learning as learning to learn with two nested loops (i.e. learning problems). The inner loop learns on each individual instance with self-supervision before final prediction. The outer loop learns the self-supervised task used by the inner loop, such that its final prediction improves. Our inner loop turns out to be equivalent to linear attention when the inner-loop learner is only a linear model, and to self-attention when it is a kernel estimator. For practical comparison with linear or self-attention layers, we replace each of them in a transformer with an inner loop, so our outer loop is equivalent to training the architecture. When each inner-loop learner is a neural network, our approach vastly outperforms transformers with linear attention on ImageNet from 224 x 224 raw pixels in both accuracy and FLOPs, while (regular) transformers cannot run.

cs.LG↗

arXiv2024

InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation. The InternVid dataset contains over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words. Our core contribution is to develop a scalable approach to autonomously build a high-quality video-text dataset with large language models (LLM), thereby showcasing its efficacy in learning video-language representation at scale. Specifically, we utilize a multi-scale approach to generate video-related descriptions. Furthermore, we introduce ViCLIP, a video-text representation learning model based on ViT-L. Learned on InternVid via contrastive learning, this model demonstrates leading zero-shot action recognition and competitive video retrieval performance. Beyond basic video understanding tasks like recognition and retrieval, our dataset and model have broad applications. They are particularly beneficial for generating interleaved video-text data for learning a video-centric dialogue system, advancing video-to-text and text-to-video generation research. These proposed resources provide a tool for researchers and practitioners interested in multimodal video understanding and generation.

cs.CV↗

arXiv2023

Electron charge qubits with 0.1 millisecond coherence time

Electron charge qubits are compelling candidates for solid-state quantum computing because of their inherent simplicity in qubit design, fabrication, control, and readout. However, all existing electron charge qubits, built upon conventional semiconductors and superconductors, suffer from severe charge noise that limits the coherence time to the order of 1 microsecond. Here, we report our experimental realization of ultralong-coherence electron charge qubits, based upon isolated single electrons trapped on an ultraclean solid neon surface in vacuum. Quantum information is encoded in the motional states of an electron that is strongly coupled with microwave photons in an on-chip superconducting resonator. The measured relaxation time $T_1$ and coherence time $T_2$ are both on the order of 0.1 milliseconds. The single-shot readout fidelity without using a quantum-limited amplifier is 98.1%. The average single-qubit gate fidelity using Clifford-based randomized benchmarking is 99.97%. Simultaneous strong coupling of two qubits with the same resonator is demonstrated, as a first step toward two-qubit entangling gates for universal quantum computing. These results manifest that the electron-on-solid-neon (eNe) charge qubits outperform all existing charge qubits to date and rival state-of-the-art superconducting transmon qubits, offering an appealing platform for quantum computing.

quant-ph↗

arXiv2023

Sum-Rate Maximization for Active RIS-Aided Downlink RSMA System

In this paper, the problem of sum-rate maximization for an active reconfigurable intelligent surface (RIS) assisted downlink rate-splitting multiple access (RSMA) transmission system is studied. In the considered model, the active RIS is deployed to overcome severe power attenuation, which is caused by the cumulative product of RIS incidence path loss and the reflection path loss. Since the active RIS can adjust both the phase and the amplitude of the incident signal simultaneously, the RIS control scheme requires delicate design to improve RSMA communication performance. To address this issue, a sum-rate maximization problem is formulated to jointly optimize the beamforming vectors, rate allocation vector, and RIS precoding matrix. To solve this non-convex sum-rate maximization problem, an iterative algorithm based on fractional programming (FP) and quadratic constraint quadratic programming (QCQP) is proposed. In particular, the proposed algorithm firstly decomposes the original problem into two subproblems, namely, 1) beamforming and rate allocation optimization and 2) active RIS precoding optimization. The corresponding variables of the two subproblems are optimized through sequential convex approximation (SCA) and block coordinate descent (BCD), respectively. Numerical results show that the proposed active RIS-aided RSMA system could increase the sum-rate by up to 45% over the conventional passive RIS-aided RSMA system with the same energy consumption.

cs.IT↗

arXiv2022

Understanding the onset of surface degradation in LiNiO2 cathodes

Nickel-based layered oxides offer an attractive platform for the development of energy-dense cobalt-free cathodes for lithium-ion batteries but suffer from degradation via oxygen gas release during electrochemical cycling. While such degradation has previously been characterized phenomenologically with experiments, an atomic-scale understanding of the reactions that take place at the cathode surface has been lacking. Here, we develop a first-principles methodology for the prediction of the surface reconstructions of intercalation electrode particles as a function of the temperature and state of charge. We report the surface phase diagrams of the LiNiO2(001) and (104) surfaces and identify surface structures that are likely visited during the first charge and discharge. Our calculations indicate that both surfaces experience oxygen loss during the first charge, resulting in irreversible changes to the surface structures. At the end of charge, the surface Ni atoms migrate into tetrahedral sites, from which they further migrate into Li vacancies during discharge, leading to Li/Ni mixed discharged surface phases. Further, the impact of the temperature and voltage range during cycling on the charge/discharge mechanism is discussed. The present study thus provides insight into the initial stages of cathode surface degradation and lies the foundation for the computational design of cathode materials that are stable against oxygen release.

cond-mat.mtrl-sci↗

arXiv2022

Single electrons on solid neon as a solid-state qubit platform

Progress toward the realization of quantum computers requires persistent advances in their constituent building blocks - qubits. Novel qubit platforms that simultaneously embody long coherence, fast operation, and large scalability offer compelling advantages in the construction of quantum computers and many other quantum information systems. Electrons, ubiquitous elementary particles of nonzero charge, spin, and mass, have commonly been perceived as paradigmatic local quantum information carriers. Despite superior controllability and configurability, their practical performance as qubits via either motional or spin states depends critically on their material environment. Here we report our experimental realization of a new qubit platform based upon isolated single electrons trapped on an ultraclean solid neon surface in vacuum. By integrating an electron trap in a circuit quantum electrodynamics architecture, we achieve strong coupling between the motional states of a single electron and a single microwave photon in an on-chip superconducting resonator. Qubit gate operations and dispersive readout are implemented to measure the energy relaxation time $T_1$ of $15~μ$s and phase coherence time $T_2$ over $200~$ns. These results indicate that the electron-on-solid-neon qubit already performs near the state of the art as a charge qubit.

quant-ph↗