arXiv subjects

Wei Li

Publications and source records attributed to Wei Li.

At least 523 records · Page 29Linked to original sources

arXiv2024

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

Creating AI systems that can interact with environments over long periods, similar to human cognition, has been a longstanding research goal. Recent advancements in multimodal large language models (MLLMs) have made significant strides in open-world understanding. However, the challenge of continuous and simultaneous streaming perception, memory, and reasoning remains largely unexplored. Current MLLMs are constrained by their sequence-to-sequence architecture, which limits their ability to process inputs and generate responses simultaneously, akin to being unable to think while perceiving. Furthermore, relying on long contexts to store historical data is impractical for long-term interactions, as retaining all information becomes costly and inefficient. Therefore, rather than relying on a single foundation model to perform all functions, this project draws inspiration from the concept of the Specialized Generalist AI and introduces disentangled streaming perception, reasoning, and memory mechanisms, enabling real-time interaction with streaming video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive (IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module: Processes multimodal information in real-time, storing key details in memory and triggering reasoning in response to user queries. (2) Multi-modal Long Memory Module: Integrates short-term and long-term memory, compressing short-term memories into long-term ones for efficient retrieval and improved accuracy. (3) Reasoning Module: Responds to queries and executes reasoning tasks, coordinating with the perception and memory modules. This project simulates human-like cognition, enabling multimodal large language models to provide continuous and adaptive service over time.

cs.CV↗

arXiv2024

HiFiVFS: High Fidelity Video Face Swapping

Face swapping aims to generate results that combine the identity from the source with attributes from the target. Existing methods primarily focus on image-based face swapping. When processing videos, each frame is handled independently, making it difficult to ensure temporal stability. From a model perspective, face swapping is gradually shifting from generative adversarial networks (GANs) to diffusion models (DMs), as DMs have been shown to possess stronger generative capabilities. Current diffusion-based approaches often employ inpainting techniques, which struggle to preserve fine-grained attributes like lighting and makeup. To address these challenges, we propose a high fidelity video face swapping (HiFiVFS) framework, which leverages the strong generative capability and temporal prior of Stable Video Diffusion (SVD). We build a fine-grained attribute module to extract identity-disentangled and fine-grained attribute features through identity desensitization and adversarial learning. Additionally, We introduce detailed identity injection to further enhance identity similarity. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) in video face swapping, both qualitatively and quantitatively.

cs.CV↗

arXiv2024

Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations

We introduce CHARM, the first benchmark for comprehensively and in-depth evaluating the commonsense reasoning ability of large language models (LLMs) in Chinese, which covers both globally known and Chinese-specific commonsense. We evaluated 7 English and 12 Chinese-oriented LLMs on CHARM, employing 5 representative prompt strategies for improving LLMs' reasoning ability, such as Chain-of-Thought. Our findings indicate that the LLM's language orientation and the task's domain influence the effectiveness of the prompt strategy, which enriches previous research findings. We built closely-interconnected reasoning and memorization tasks, and found that some LLMs struggle with memorizing Chinese commonsense, affecting their reasoning ability, while others show differences in reasoning despite similar memorization performance. We also evaluated the LLMs' memorization-independent reasoning abilities and analyzed the typical errors. Our study precisely identified the LLMs' strengths and weaknesses, providing the clear direction for optimization. It can also serve as a reference for studies in other fields. We will release CHARM at https://github.com/opendatalab/CHARM .

cs.CL↗

arXiv2024

Identifying percolation phase transitions with unsupervised learning based on largest clusters

The application of machine learning in the study of phase transitions has achieved remarkable success in both equilibrium and non-equilibrium systems. It is widely recognized that unsupervised learning can retrieve phase transition information through hidden variables. However, using unsupervised methods to identify the critical point of percolation models has remained an intriguing challenge. This paper suggests that, by inputting the largest cluster rather than the original configuration into the learning model, unsupervised learning can indeed predict the critical point of the percolation model. Furthermore, we observe that when the largest cluster configuration is randomly shuffled-altering the positions of occupied sites or bonds-there is no significant difference in the output compared to learning the largest cluster configuration directly. This finding suggests a more general principle: unsupervised learning primarily captures particle density, or more specifically, occupied site density. However, shuffling does impact the formation of the largest cluster, which is directly related to phase transitions. As randomness increases, we observe that the correlation length tends to decrease, providing direct evidence of this relationship. We also propose a method called Fake Finite Size Scaling (FFSS) to calculate the critical value, which improves the accuracy of fitting to a great extent.

cond-mat.stat-mech↗

arXiv2024

Double Magnon-Roton Excitations in the Triangular-Lattice Spin Supersolid

Supersolid is an exotic quantum state of matter that spontaneously hosts the features of both solid and superfluid, which breaks the translation and U(1) gauge symmetries. Here we study the spin dynamics in the triangular-lattice compound Na$_2$BaCo(PO$_4$)$_2$, which is revealed in [Xiang et al., Nature 625, 270-275 (2024)] as a quantum magnetic analog of supersolid. We simulate the easy-axis Heisenberg model with tensor network approach and uncover unique dynamic traits. These features are manifested in two branches of excitations that can be associated with the spin solidity and superfluidity, respectively. One branch contains the U(1) Goldstone and roton modes, while the other comprises pseudo-Goldstone and roton modes. The gapless Goldstone modes of the in-plane superfluid order are confirmed by our inelastic neutron scattering measurements. Together with the evident out-of-plane solid order indicated by the magnetic Bragg peaks, our findings provide spectroscopic evidence for spin supersolidity in this easy-axis antiferromagnet. Akin to the role of phonon-roton modes -- Landau elementary excitations -- in shaping the helium superfluid thermodynamics, the intriguing double magnon-roton dispersion here determines the low-temperature thermodynamics of spin supersolid down to sub-Kelvin regime, explaining the recently observed giant magnetocaloric effect in Na$_2$BaCo(PO$_4$)$_2$.

cond-mat.str-el↗

arXiv2024

Successive magnetic transitions in the spin-5/2 easy-axis triangular-lattice antiferromagnet Na$_2$BaMn(PO$_4$)$_2$: A neutron diffraction study

Motivated by the recent observations of various exotic quantum states in the equilateral triangular-lattice phosphates Na$_2$BaCo(PO$_4$)$_2$ with $J\rm_{eff}$ = 1/2 and Na$_2$BaNi(PO$_4$)$_2$ with $S$ = 1, the magnetic properties of spin-5/2 antiferromagnet Na$_2$BaMn(PO$_4$)$_2$, their classical counterpart, are comprehensively investigated experimentally. DC magnetization and specific heat measurements on polycrystalline samples indicate two successive magnetic transitions at $T\rm_{N1}$ $\approx$ 1.13 K and $T\rm_{N2}$ $\approx$ 1.28 K, respectively. Zero-field neutron powder diffraction measurement at 67 mK reveals a Y-like spin configuration as its ground-state magnetic structure, with both the $ab$-plane and $c$-axis components of the Mn$^{2+}$ moments long-range ordered. The incommensurate magnetic propagation vector $k$ shows a dramatic change for the intermediate phase between $T\rm_{N1}$ and $T\rm_{N2}$, in which the spin state is speculated to change into a collinear structure with only the $c$-axis moments ordered, as stabilized by thermal fluctuations. The successive magnetic transitions observed in Na$_2$BaMn(PO$_4$)$_2$ are in line with the expectation for a triangle-lattice antiferromagnet with an easy-axis magnetic anisotropy.

cond-mat.str-el↗

arXiv2024

FD-LLM: Large Language Model for Fault Diagnosis of Machines

Large language models (LLMs) are effective at capturing complex, valuable conceptual representations from textual data for a wide range of real-world applications. However, in fields like Intelligent Fault Diagnosis (IFD), incorporating additional sensor data-such as vibration signals, temperature readings, and operational metrics-is essential but it is challenging to capture such sensor data information within traditional text corpora. This study introduces a novel IFD approach by effectively adapting LLMs to numerical data inputs for identifying various machine faults from time-series sensor data. We propose FD-LLM, an LLM framework specifically designed for fault diagnosis by formulating the training of the LLM as a multi-class classification problem. We explore two methods for encoding vibration signals: the first method uses a string-based tokenization technique to encode vibration signals into text representations, while the second extracts statistical features from both the time and frequency domains as statistical summaries of each signal. We assess the fault diagnosis capabilities of four open-sourced LLMs based on the FD-LLM framework, and evaluate the models' adaptability and generalizability under various operational conditions and machine components, namely for traditional fault diagnosis, cross-operational conditions, and cross-machine component settings. Our results show that LLMs such as Llama3 and Llama3-instruct demonstrate strong fault detection capabilities and significant adaptability across different operational conditions, outperforming state-of-the-art deep learning (DL) approaches in many cases.

cs.AI↗

arXiv2024

The behavior of rich-club coefficient in scale-free networks

The rich-club phenomenon, which provides information about the association between nodes, is a useful method to study the hierarchy structure of networks. In this work, we explore the behavior of rich-club coefficient (RCC) in scale-free networks, and find that the degree-based RCC is a power function of degree centrality with power exponent $β$ and the betweenness-based RCC is a linear function of betweenness centrality with slope $θ$. Moreover, we calculate the value of RCC in a BA network by deleting nodes and obtain a general expression for RCC as a function of node sequence. On this basis, the solution of RCC for centrality is also obtained, which shows that the curve of RCC is determined by the centrality distribution. In the numerical simulation, we observe: $β$ and $γ$ (the degree distribution exponent) increase together, $θ$ increases with the average degree and decreases to convergence as $γ$ increases.

physics.soc-ph↗

arXiv2024

Phase Diagram, $d$-Wave Superconductivity, and Pseudogap of the $t$-$t'$-$J$ Model at Finite Temperature

Recently, robust $d$-wave superconductive (SC) order has been unveiled in the ground state of the 2D $t$-$t'$-$J$ model -- with both nearest-neighbor ($t$) and next-nearest-neighbor ($t'$) hoppings -- by density matrix renormalization group studies. However, there is currently a debate on whether the $d$-wave SC holds up strong on both $t'/t>0$ and $t'/t<0$ cases for the $t$-$t'$-$J$ model, which correspond to the electron- and hole-doped sides of the cuprate phase diagram, respectively. Here we exploit state-of-the-art thermal tensor network approach to accurately obtain the phase diagram of the $t$-$t'$-$J$ model on cylinders with widths up to $W=6$ and down to low temperature as $T/J \simeq 0.06$, pushing the boundaries of contemporary finite-$T$ calculations. For $t'/t>0$, we find a dome-like SC regime with a diverging $d$-wave pairing susceptibility, $χ_\textrm{SC} \propto 1/T^α$ below a characteristic temperature $T_c^*$. Near optimal doping, $T_c^*$ reaches its highest value of about $0.15 J$. Above $T_c^*$ yet below a higher crossover temperature $T^*$, the magnetic susceptibility becomes suppressed, which can be related to the onset of pseudogap (PG) behaviors. On the other hand, for $t'/t<0$ we find the pairing correlations are much weaker, although there exhibits a node-antinode structure in the PG regime as observed in the hole-doped cuprates. The thermal tensor network calculations of the $t$-$t'$-$J$ model underscore both the similarities and differences in the finite-temperature phase diagram between the fundamental model and cuprates, yielding unique insights into their intricate behaviors.

cond-mat.str-el↗

arXiv2024

MM-SEAL: A Large-scale Video Dataset of Multi-person Multi-grained Spatio-temporally Action Localization

In this paper, we introduce a novel large-scale video dataset dubbed MM-SEAL for multi-person multi-grained spatio-temporal action localization among human daily life. We are the first to propose a new benchmark for multi-person spatio-temporal complex activity localization, where complex semantic and long duration bring new challenges to localization tasks. We observe that limited atomic actions can be combined into many complex activities. MM-SEAL provides both atomic action and complex activity annotations, producing 111.7k atomic actions spanning 172 action categories and 17.7k complex activities spanning 200 activity categories. We explore the relationship between atomic actions and complex activities, finding that atomic action features can improve the complex activity localization performance. Also, we propose a new network which generates temporal proposals and labels simultaneously, termed Faster-TAD. Finally, our evaluations show that visual features pretrained on MM-SEAL can improve the performance on other action localization benchmarks. We will release the dataset and the project code upon publication of the paper.

cs.CV↗

arXiv2024

OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining

Surgical practice involves complex visual interpretation, procedural skills, and advanced medical knowledge, making surgical vision-language pretraining (VLP) particularly challenging due to this complexity and the limited availability of annotated data. To address the gap, we propose OphCLIP, a hierarchical retrieval-augmented vision-language pretraining framework specifically designed for ophthalmic surgical workflow understanding. OphCLIP leverages the OphVL dataset we constructed, a large-scale and comprehensive collection of over 375K hierarchically structured video-text pairs with tens of thousands of different combinations of attributes (surgeries, phases/operations/actions, instruments, medications, as well as more advanced aspects like the causes of eye diseases, surgical objectives, and postoperative recovery recommendations, etc). These hierarchical video-text correspondences enable OphCLIP to learn both fine-grained and long-term visual representations by aligning short video clips with detailed narrative descriptions and full videos with structured titles, capturing intricate surgical details and high-level procedural insights, respectively. Our OphCLIP also designs a retrieval-augmented pretraining framework to leverage the underexplored large-scale silent surgical procedure videos, automatically retrieving semantically relevant content to enhance the representation learning of narrative videos. Evaluation across 11 datasets for phase recognition and multi-instrument identification shows OphCLIP's robust generalization and superior performance.

cs.CV↗

arXiv2024

MRIFE: A Mask-Recovering and Interactive-Feature-Enhancing Semantic Segmentation Network For Relic Landslide Detection

Relic landslide, formed over a long period, possess the potential for reactivation, making them a hazardous geological phenomenon. While reliable relic landslide detection benefits the effective monitoring and prevention of landslide disaster, semantic segmentation using high-resolution remote sensing images for relic landslides faces many challenges, including the object visual blur problem, due to the changes of appearance caused by prolonged natural evolution and human activities, and the small-sized dataset problem, due to difficulty in recognizing and labelling the samples. To address these challenges, a semantic segmentation model, termed mask-recovering and interactive-feature-enhancing (MRIFE), is proposed for more efficient feature extraction and separation. Specifically, a contrastive learning and mask reconstruction method with locally significant feature enhancement is proposed to improve the ability to distinguish between the target and background and represent landslide semantic features. Meanwhile, a dual-branch interactive feature enhancement architecture is used to enrich the extracted features and address the issue of visual ambiguity. Self-distillation learning is introduced to leverage the feature diversity both within and between samples for contrastive learning, improving sample utilization, accelerating model convergence, and effectively addressing the problem of the small-sized dataset. The proposed MRIFE is evaluated on a real relic landslide dataset, and experimental results show that it greatly improves the performance of relic landslide detection. For the semantic segmentation task, compared to the baseline, the precision increases from 0.4226 to 0.5347, the mean intersection over union (IoU) increases from 0.6405 to 0.6680, the landslide IoU increases from 0.3381 to 0.3934, and the F1-score increases from 0.5054 to 0.5646.

cs.CV↗

arXiv2024

Fact-Level Confidence Calibration and Self-Correction

Confidence calibration in LLMs, i.e., aligning their self-assessed confidence with the actual accuracy of their responses, enabling them to self-evaluate the correctness of their outputs. However, current calibration methods for LLMs typically estimate two scalars to represent overall response confidence and correctness, which is inadequate for long-form generation where the response includes multiple atomic facts and may be partially confident and correct. These methods also overlook the relevance of each fact to the query. To address these challenges, we propose a Fact-Level Calibration framework that operates at a finer granularity, calibrating confidence to relevance-weighted correctness at the fact level. Furthermore, comprehensive analysis under the framework inspired the development of Confidence-Guided Fact-level Self-Correction ($\textbf{ConFix}$), which uses high-confidence facts within a response as additional knowledge to improve low-confidence ones. Extensive experiments across four datasets and six models demonstrate that ConFix effectively mitigates hallucinations without requiring external knowledge sources such as retrieval systems.

cs.CL↗

arXiv2024

On the Effects of Data Scale on UI Control Agents

Autonomous agents that control computer interfaces to accomplish human tasks are emerging. Leveraging LLMs to power such agents has been of special interest, but unless fine-tuned on human-collected task demonstrations, performance is still relatively low. In this work we study whether fine-tuning alone is a viable approach for building real-world computer control agents. In particularly, we investigate how performance measured on both high and low-level tasks in domain and out of domain scales as more training data is collected. To this end we collect and release a new dataset, AndroidControl, consisting of 15,283 demonstrations of everyday tasks with Android apps. Compared to existing datasets, each AndroidControl task instance includes both high and low-level human-generated instructions, allowing us to explore the level of task complexity an agent can handle. Moreover, AndroidControl is the most diverse computer control dataset to date, including 14,548 unique tasks over 833 Android apps, thus allowing us to conduct in-depth analysis of the model performance in and out of the domain of the training data. Using the dataset, we find that when tested in domain fine-tuned models outperform zero and few-shot baselines and scale in such a way that robust performance might feasibly be obtained simply by collecting more data. Out of domain, performance scales significantly more slowly and suggests that in particular for high-level tasks, fine-tuning on more data alone may be insufficient for achieving robust out-of-domain performance.

cs.AI↗

arXiv2024

Tuning the Quasi-bound States of Double-barrier Structures: Insights from Resonant Tunneling Spectra

In this work, we study the resonant tunneling (RT) of electrons and H atoms in double-barrier (DB) systems. Our numerical calculations directly verify the correspondence between the resonant tunneling energies and the energy levels of quasi-bound states (QBS) within the double barriers. Based on this, in-depth analyses are carried out on the modulation of QBS energy levels and numbers which show step variation with the inter-barrier spacing. The mathematical criterion for the existence of QBS is derived, and the impacts of the barrier width and barrier height on QBS levels are investigated. Taking the rectangular double-barrier as an example, we have studied the manipulation of electronic structures and optical properties of the inter-barrier region (quasi-potential well) by tuning the inter-barrier spacing (width of quasi-potential well). Atom-like optical absorption features are found in the range of infrared to visible spectrum, which can be continuously tuned by the variation of quasi-potential well width. The potential application of double-barrier nanostructures in ultrahigh-precision detection of electromagnetic radiations is demonstrated.

cond-mat.mes-hall↗

arXiv2024

UNSCT-HRNet: Modeling Anatomical Uncertainty for Landmark Detection in Total Hip Arthroplasty

Total hip arthroplasty (THA) relies on accurate landmark detection from radiographic images, but unstructured data caused by irregular patient postures or occluded anatomical markers pose significant challenges for existing methods. To address this, we propose UNSCT-HRNet (Unstructured CT - High-Resolution Net), a deep learning-based framework that integrates a Spatial Relationship Fusion (SRF) module and an Uncertainty Estimation (UE) module. The SRF module, utilizing coordinate convolution and polarized attention, enhances the model's ability to capture complex spatial relationships. Meanwhile, the UE module which based on entropy ensures predictions are anatomically relevant. For unstructured data, the proposed method can predict landmarks without relying on the fixed number of points, which shows higher accuracy and better robustness comparing with the existing methods. Our UNSCT-HRNet demonstrates over a 60% improvement across multiple metrics in unstructured data. The experimental results also reveal that our approach maintains good performance on the structured dataset. Overall, the proposed UNSCT-HRNet has the potential to be used as a new reliable, automated solution for THA surgical planning and postoperative monitoring.

eess.IV↗

arXiv2024

Physics-Informed Neural Networks with Complementary Soft and Hard Constraints for Solving Complex Boundary Navier-Stokes Equations

Soft- and hard-constrained Physics Informed Neural Networks (PINNs) have achieved great success in solving partial differential equations (PDEs). However, these methods still face great challenges when solving the Navier-Stokes equations (NSEs) with complex boundary conditions. To address these challenges, this paper introduces a novel complementary scheme combining soft and hard constraint PINN methods. The soft-constrained part is thus formulated to obtain the preliminary results with a lighter training burden, upon which refined results are then achieved using a more sophisticated hard-constrained mechanism with a primary network and a distance metric network. Specifically, the soft-constrained part focuses on boundary points, while the primary network emphasizes inner domain points, primarily through PDE loss. Additionally, the novel distance metric network is proposed to predict the power function of the distance from a point to the boundaries, which serves as the weighting factor for the first two components. This approach ensures accurate predictions for both boundary and inner domain areas. The effectiveness of the proposed method on the NSEs problem with complex boundary conditions is demonstrated by solving a 2D cylinder wake problem and a 2D blocked cavity flow with a segmented inlet problem, achieving significantly higher accuracy compared to traditional soft- and hard-constrained PINN approaches. Given PINN's inherent advantages in solving the inverse and the large-scale problems, which are challenging for traditional computational fluid dynamics (CFD) methods, this approach holds promise for the inverse design of required flow fields by specifically-designed boundary conditions and the reconstruction of large-scale flow fields by adding a limited number of training input points. The code for our approach will be made publicly available.

physics.flu-dyn↗

arXiv2024

The tricritical point of tricritical directed percolation is determined based on neural network

In recent years, neural networks have increasingly been employed to identify critical points of phase transitions. For the tricritical directed percolation model, its steady-state configurations encompass both first-order and second-order phase transitions. Due to the presence of crossover effects, identifying the critical points of phase transitions becomes challenging. This study utilizes Monte Carlo simulations to obtain steady-state configurations under different probabilities $p$ and $q$, and by calculating the increments in average particle density, we observe first-order transitions, second-order transitions, and regions where both types of transitions interact.These Monte Carlo-generated steady-state configurations are used as input to construct and train a convolutional neural network, from which we determine the critical points $p_{c}$ for different probabilities $q$. Furthermore, by learning the steady-state configurations associated with the superheated point $p=p_u$, we locate the tricritical point at $q_{t}=0.893$. Simultaneously, we employed a three-output CNN model to obtain the phase transition boundaries and the range of the crossover regions. Our method offers a neural network-based approach to capture critical points and distinguish phase transition boundaries, providing a novel solution to this problem.

cond-mat.stat-mech↗