arXiv subjects

Wei Li

Publications and source records attributed to Wei Li.

At least 541 records · Page 30Linked to original sources

arXiv2024

Sub-DM:Subspace Diffusion Model with Orthogonal Decomposition for MRI Reconstruction

Diffusion model-based approaches recently achieved re-markable success in MRI reconstruction, but integration into clinical routine remains challenging due to its time-consuming convergence. This phenomenon is partic-ularly notable when directly apply conventional diffusion process to k-space data without considering the inherent properties of k-space sampling, limiting k-space learning efficiency and image reconstruction quality. To tackle these challenges, we introduce subspace diffusion model with orthogonal decomposition, a method (referred to as Sub-DM) that restrict the diffusion process via projections onto subspace as the k-space data distribution evolves toward noise. Particularly, the subspace diffusion model circumvents the inference challenges posed by the com-plex and high-dimensional characteristics of k-space data, so the highly compact subspace ensures that diffusion process requires only a few simple iterations to produce accurate prior information. Furthermore, the orthogonal decomposition strategy based on wavelet transform hin-ders the information loss during the migration of the vanilla diffusion process to the subspace. Considering the strate-gy is approximately reversible, such that the entire pro-cess can be reversed. As a result, it allows the diffusion processes in different spaces to refine models through a mutual feedback mechanism, enabling the learning of ac-curate prior even when dealing with complex k-space data. Comprehensive experiments on different datasets clearly demonstrate that the superiority of Sub-DM against state of-the-art methods in terms of reconstruction speed and quality.

eess.IV↗

arXiv2024

TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment

Recent advancements in image understanding have benefited from the extensive use of web image-text pairs. However, video understanding remains a challenge despite the availability of substantial web video-text data. This difficulty primarily arises from the inherent complexity of videos and the inefficient language supervision in recent web-collected video-text datasets. In this paper, we introduce Text-Only Pre-Alignment (TOPA), a novel approach to extend large language models (LLMs) for video understanding, without the need for pre-training on real video data. Specifically, we first employ an advanced LLM to automatically generate Textual Videos comprising continuous textual frames, along with corresponding annotations to simulate real video-text data. Then, these annotated textual videos are used to pre-align a language-only LLM with the video modality. To bridge the gap between textual and real videos, we employ the CLIP model as the feature extractor to align image and text modalities. During text-only pre-alignment, the continuous textual frames, encoded as a sequence of CLIP text features, are analogous to continuous CLIP image features, thus aligning the LLM with real video representation. Extensive experiments, including zero-shot evaluation and finetuning on various video understanding tasks, demonstrate that TOPA is an effective and efficient framework for aligning video content with LLMs. In particular, without training on any video data, the TOPA-Llama2-13B model achieves a Top-1 accuracy of 51.0% on the challenging long-form video understanding benchmark, Egoschema. This performance surpasses previous video-text pre-training approaches and proves competitive with recent GPT-3.5-based video agents.

cs.CV↗

arXiv2024

Identifying and bounding the probability of necessity for causes of effects with ordinal outcomes

Although the existing causal inference literature focuses on the forward-looking perspective by estimating effects of causes, the backward-looking perspective can provide insights into causes of effects. In backward-looking causal inference, the probability of necessity measures the probability that a certain event is caused by the treatment given the observed treatment and outcome. Most existing results focus on binary outcomes. Motivated by applications with ordinal outcomes, we propose a general definition of the probability of necessity. However, identifying the probability of necessity is challenging because it involves the joint distribution of the potential outcomes. We propose a novel assumption of monotonic incremental treatment effect to identify the probability of necessity with ordinal outcomes. We also discuss the testable implications of this key identification assumption. When it fails, we derive explicit formulas of the sharp large-sample bounds on the probability of necessity.

math.ST↗

arXiv2024

Make Continual Learning Stronger via C-Flat

Model generalization ability upon incrementally acquiring dynamically updating knowledge from sequentially arriving tasks is crucial to tackle the sensitivity-stability dilemma in Continual Learning (CL). Weight loss landscape sharpness minimization seeking for flat minima lying in neighborhoods with uniform low loss or smooth gradient is proven to be a strong training regime improving model generalization compared with loss minimization based optimizer like SGD. Yet only a few works have discussed this training regime for CL, proving that dedicated designed zeroth-order sharpness optimizer can improve CL performance. In this work, we propose a Continual Flatness (C-Flat) method featuring a flatter loss landscape tailored for CL. C-Flat could be easily called with only one line of code and is plug-and-play to any CL methods. A general framework of C-Flat applied to all CL categories and a thorough comparison with loss minima optimizer and flat minima based CL approaches is presented in this paper, showing that our method can boost CL performance in almost all cases. Code is available at https://github.com/WanNaa/C-Flat.

cs.LG↗

arXiv2024

Pseudo-hermitian Chebyshev differential matrix and non-Hermitian Liouville quantum mechanics

The spectral collocation method (SCM) exhibits a clear superiority in solving ordinary and partial differential equations compared to conventional techniques, such as finite difference and finite element methods. This makes SCM a powerful tool for addressing the Schrödinger-like equations with boundary conditions in physics. However, the Chebyshev differential matrix (CDM), commonly used in SCM to replace the differential operator, is not Hermitian but pseudo-Hermitian. This non-Hermiticity subtly affects the pseudospectra and leads to a loss of completeness in the eigenstates. Consequently, several issues arise with these eigenstates. In this paper, we revisit the non-Hermitian Liouville quantum mechanics by emphasizing the pseudo-Hermiticity of the CDM and explore its expanded models. Furthermore, we demonstrate that the spectral instability can be influenced by the compactification parameter.

quant-ph↗

arXiv2024

Restoring Kibble-Zurek Scaling and Defect Freezing in Non-Hermitian Systems under Biorthogonal Framework

Non-Hermitian physics provides an effective description of open and nonequilibrium systems and hosts many novel and intriguing phenomena such as exceptional points and non-Hermitian skin effect. Despite extensive theoretical and experimental studies, however, how to properly deal with the nonadiabatic dynamics in driven non-Hermitian quantum system is still under debate. Here, we develop a theoretical framework based on time-dependent biorthogonal quantum formalism by redefining the associated state to obtain the gauge-independent transition probability, and study the nonadiabatic dynamics of a linearly driven non-Hermitian system. In contrast to the normalization method that leads to a modified Kibble-Zurek scaling behavior, our approach predicts that the defect production at exceptional points exhibits power-law scaling behaviors conforming to the Kibble-Zurek mechanism. In the fast quench regime, universal scaling behaviors are also found with respect to the initial quenching parameter, which can be explained by the impulse-adiabatic approximation. Moreover, as trespassing the PT -broken region, the phenomenon of defect freezing, i.e., violation of adiabaticity, is observed.

quant-ph↗

arXiv2024

Technical Report for SoccerNet Challenge 2022 -- Replay Grounding Task

In order to make full use of video information, we transform the replay grounding problem into a video action location problem. We apply a unified network Faster-TAD proposed by us for temporal action detection to get the results of replay grounding. Finally, by observing the data distribution of the training data, we refine the output of the model to get the final submission.

cs.CV↗

arXiv2024

Technical Report for Soccernet 2023 -- Dense Video Captioning

In the task of dense video captioning of Soccernet dataset, we propose to generate a video caption of each soccer action and locate the timestamp of the caption. Firstly, we apply Blip as our video caption framework to generate video captions. Then we locate the timestamp by using (1) multi-size sliding windows (2) temporal proposal generation and (3) proposal classification.

cs.CV↗

arXiv2024

Technical Report for ActivityNet Challenge 2022 -- Temporal Action Localization

In the task of temporal action localization of ActivityNet-1.3 datasets, we propose to locate the temporal boundaries of each action and predict action class in untrimmed videos. We first apply VideoSwinTransformer as feature extractor to extract different features. Then we apply a unified network following Faster-TAD to simultaneously obtain proposals and semantic labels. Last, we ensemble the results of different temporal action detection models which complement each other. Faster-TAD simplifies the pipeline of TAD and gets remarkable performance, obtaining comparable results as those of multi-step approaches.

cs.CV↗

arXiv2024

CopRA: A Progressive LoRA Training Strategy

Low-Rank Adaptation (LoRA) is a parameter-efficient technique for rapidly fine-tuning foundation models. In standard LoRA training dynamics, models tend to quickly converge to a local optimum near the initialization. However, this local optimum may not be ideal for out-of-distribution data or tasks such as merging and pruning. In this work, we propose a novel progressive training strategy for LoRA with random layer dropping. This strategy also optimizes the Shapley value of LoRA parameters in each layer, treating each layer as a player in a cooperative game. We refer to this method as Cooperative LoRA (CopRA). Our experimental results demonstrate that parameters trained with CopRA exhibit linear mode connectivity, which enables efficient model merging. This also paves the way for federated learning and multi-task learning via LoRA merging. Additionally, by optimizing the Shapley value, CopRA shows superior performance in pruning tasks.

cs.LG↗

arXiv2024

Einstein Probe discovery of EP240408a: a peculiar X-ray transient with an intermediate timescale

We report the discovery of a peculiar X-ray transient, EP240408a, by Einstein Probe (EP) and follow-up studies made with EP, Swift, NICER, GROND, ATCA and other ground-based multi-wavelength telescopes. The new transient was first detected with Wide-field X-ray Telescope (WXT) on board EP on April 8th, 2024, manifested in an intense yet brief X-ray flare lasting for 12 seconds. The flare reached a peak flux of 3.9x10^(-9) erg/cm2/s in 0.5-4 keV, about 300 times brighter than the underlying X-ray emission detected throughout the observation. Rapid and more precise follow-up observations by EP/FXT, Swift and NICER confirmed the finding of this new transient. Its X-ray spectrum is non-thermal in 0.5-10 keV, with a power-law photon index varying within 1.8-2.5. The X-ray light curve shows a plateau lasting for about 4 days, followed by a steep decay till becoming undetectable about 10 days after the initial detection. Based on its temporal property and constraints from previous EP observations, an unusual timescale in the range of 7-23 days is found for EP240408a, which is intermediate between the commonly found fast and long-term transients. No counterparts have been found in optical and near-infrared, with the earliest observation at 17 hours after the initial X-ray detection, suggestive of intrinsically weak emission in these bands. We demonstrate that the remarkable properties of EP240408a are inconsistent with any of the transient types known so far, by comparison with, in particular, jetted tidal disruption events, gamma-ray bursts, X-ray binaries and fast blue optical transients. The nature of EP240408a thus remains an enigma. We suggest that EP240408a may represent a new type of transients with intermediate timescales of the order of about 10 days. The detection and follow-ups of more of such objects are essential for revealing their origin.

astro-ph.HE↗

arXiv2024

Pulling Target to Source: A New Perspective on Domain Adaptive Semantic Segmentation

Domain adaptive semantic segmentation aims to transfer knowledge from a labeled source domain to an unlabeled target domain. However, existing methods primarily focus on directly learning qualified target features, making it challenging to guarantee their discrimination in the absence of target labels. This work provides a new perspective. We observe that the features learned with source data manage to keep categorically discriminative during training, thereby enabling us to implicitly learn adequate target representations by simply \textbf{pulling target features close to source features for each category}. To this end, we propose T2S-DA, which we interpret as a form of pulling Target to Source for Domain Adaptation, encouraging the model in learning similar cross-domain features. Also, considering the pixel categories are heavily imbalanced for segmentation datasets, we come up with a dynamic re-weighting strategy to help the model concentrate on those underperforming classes. Extensive experiments confirm that T2S-DA learns a more discriminative and generalizable representation, significantly surpassing the state-of-the-art. We further show that our method is quite qualified for the domain generalization task, verifying its domain-invariant property.

cs.CV↗

arXiv2024

PGDiffSeg: Prior-Guided Denoising Diffusion Model with Parameter-Shared Attention for Breast Cancer Segmentation

Early detection through imaging and accurate diagnosis is crucial in mitigating the high mortality rate associated with breast cancer. However, locating tumors from low-resolution and high-noise medical images is extremely challenging. Therefore, this paper proposes a novel PGDiffSeg (Prior-Guided Diffusion Denoising Model with Parameter-Shared Attention) that applies diffusion denoising methods to breast cancer medical image segmentation, accurately recovering the affected areas from Gaussian noise. Firstly, we design a parallel pipeline for noise processing and semantic information processing and propose a parameter-shared attention module (PSA) in multi-layer that seamlessly integrates these two pipelines. This integration empowers PGDiffSeg to incorporate semantic details at multiple levels during the denoising process, producing highly accurate segmentation maps. Secondly, we introduce a guided strategy that leverages prior knowledge to simulate the decision-making process of medical professionals, thereby enhancing the model's ability to locate tumor positions precisely. Finally, we provide the first-ever discussion on the interpretability of the generative diffusion model in the context of breast cancer segmentation. Extensive experiments have demonstrated the superiority of our model over the current state-of-the-art approaches, confirming its effectiveness as a flexible diffusion denoising method suitable for medical image research. Our code will be publicly available later.

eess.IV↗

arXiv2024

Autoencoder-assisted study of directed percolation with spatial long-range interactions

Spatial L{é}vy-like flights are introduced as a way in the absorbing phase transitions to produce non-local interactions. We utilize the autoencoder, an unsupervised learning method, to predict the critical points for $(1+1)$-d directed percolation with such spatial long-range interactions. After making a global coverage of the reaction-diffusion distance and taking a series of different values for the parameter $β$ in the distribution $P(r){\sim}1/r^β$, the critical points $P_c$ that can be continuously varied are obtained. And the dynamic decay of the particle density under the critical points was counted as a way to determine the critical exponent $δ$ of the survival rate. We also investigate the active behavior of the system's particles under the critical point with increasing time steps, which allows us to determine the characteristic time $t_f$ of the finite-scale systems. And the dynamic exponents $z$ are obtained using the scaling relation $t_f{\sim}L^{z}$. We find that the autoencoder can identify this characteristic evolutionary behavior of particles. Finally, we discuss the compliance of the scaling form $1/δ-(β-2)/δz=2$ in different $β$ intervals as well as a method to introduce a global scaling mechanism by generating a random walking step using the L{é}vy distribution.

cond-mat.stat-mech↗

arXiv2024

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 53.96%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI.

eess.IV↗

arXiv2024

Beyond Inter-Item Relations: Dynamic Adaption for Enhancing LLM-Based Sequential Recommendation

Sequential recommender systems (SRS) predict the next items that users may prefer based on user historical interaction sequences. Inspired by the rise of large language models (LLMs) in various AI applications, there is a surge of work on LLM-based SRS. Despite their attractive performance, existing LLM-based SRS still exhibit some limitations, including neglecting intra-item relations, ignoring long-term collaborative knowledge and using inflexible architecture designs for adaption. To alleviate these issues, we propose an LLM-based sequential recommendation model named DARec. Built on top of coarse-grained adaption for capturing inter-item relations, DARec is further enhanced with (1) context masking that models intra-item relations to help LLM better understand token and item semantics in the context of SRS, (2) collaborative knowledge injection that helps LLM incorporate long-term collaborative knowledge, and (3) a dynamic adaption mechanism that uses Bayesian optimization to flexibly choose layer-wise adapter architectures in order to better incorporate different sequential information. Extensive experiments demonstrate that DARec can effectively handle sequential recommendation in a dynamic and adaptive manner.

cs.IR↗

arXiv2024

Hi-Mamba: Hierarchical Mamba for Efficient Image Super-Resolution

State Space Models (SSM), such as Mamba, have shown strong representation ability in modeling long-range dependency with linear complexity, achieving successful applications from high-level to low-level vision tasks. However, SSM's sequential nature necessitates multiple scans in different directions to compensate for the loss of spatial dependency when unfolding the image into a 1D sequence. This multi-direction scanning strategy significantly increases the computation overhead and is unbearable for high-resolution image processing. To address this problem, we propose a novel Hierarchical Mamba network, namely, Hi-Mamba, for image super-resolution (SR). Hi-Mamba consists of two key designs: (1) The Hierarchical Mamba Block (HMB) assembled by a Local SSM (L-SSM) and a Region SSM (R-SSM) both with the single-direction scanning, aggregates multi-scale representations to enhance the context modeling ability. (2) The Direction Alternation Hierarchical Mamba Group (DA-HMG) allocates the isomeric single-direction scanning into cascading HMBs to enrich the spatial relationship modeling. Extensive experiments demonstrate the superiority of Hi-Mamba across five benchmark datasets for efficient SR. For example, Hi-Mamba achieves a significant PSNR improvement of 0.29 dB on Manga109 for $\times3$ SR, compared to the strong lightweight MambaIR.

cs.CV↗

arXiv2024

Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization

Videos contain a wealth of information, and generating detailed and accurate descriptions in natural language is a key aspect of video understanding. In this paper, we present video-SALMONN 2, an advanced audio-visual large language model (LLM) with low-rank adaptation (LoRA) designed for enhanced video (with paired audio) captioning through directed preference optimization (DPO). We propose new metrics to evaluate the completeness and accuracy of video descriptions, which are optimized using DPO. To further improve training, we introduce a novel multi-round DPO (mrDPO) approach, which involves periodically updating the DPO reference model, merging and re-initializing the LoRA module as a proxy for parameter updates after each training round (1,000 steps), and incorporating guidance from ground-truth video captions to stabilize the process. To address potential catastrophic forgetting of non-captioning abilities due to mrDPO, we propose rebirth tuning, which finetunes the pre-DPO LLM by using the captions generated by the mrDPO-trained model as supervised labels. Experiments show that mrDPO significantly enhances video-SALMONN 2's captioning accuracy, reducing global and local error rates by 40\% and 20\%, respectively, while decreasing the repetition rate by 35\%. The final video-SALMONN 2 model, with just 7 billion parameters, surpasses leading models such as GPT-4o and Gemini-1.5-Pro in video captioning tasks, while maintaining competitive performance to the state-of-the-art on widely used video question-answering benchmark among models of similar size. Upon acceptance, we will release the code, model checkpoints, and training and test data. Demos are available at \href{https://video-salmonn-2.github.io}{https://video-salmonn-2.github.io}.

cs.CV↗