arXiv subjects

Feng Liu

Publications and source records attributed to Feng Liu.

At least 325 records · Page 18Linked to original sources

arXiv2025

Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding

Applying Multimodal Large Language Models (MLLMs) to video understanding presents significant challenges due to the need to model temporal relations across frames. Existing approaches adopt either implicit temporal modeling, relying solely on the LLM decoder, or explicit temporal modeling, employing auxiliary temporal encoders. To investigate this debate between the two paradigms, we propose the Stackable Temporal Encoder (STE). STE enables flexible explicit temporal modeling with adjustable temporal receptive fields and token compression ratios. Using STE, we systematically compare implicit and explicit temporal modeling across dimensions such as overall performance, token compression effectiveness, and temporal-specific understanding. We also explore STE's design considerations and broader impacts as a plug-in module and in image modalities. Our findings emphasize the critical role of explicit temporal modeling, providing actionable insights to advance video MLLMs.

cs.CV↗

arXiv2025

Lossy Compression with Pretrained Diffusion Models

We apply the DiffC algorithm (Theis et al. 2022) to Stable Diffusion 1.5, 2.1, XL, and Flux-dev, and demonstrate that these pretrained models are remarkably capable lossy image compressors. A principled algorithm for lossy compression using pretrained diffusion models has been understood since at least Ho et al. 2020, but challenges in reverse-channel coding have prevented such algorithms from ever being fully implemented. We introduce simple workarounds that lead to the first complete implementation of DiffC, which is capable of compressing and decompressing images using Stable Diffusion in under 10 seconds. Despite requiring no additional training, our method is competitive with other state-of-the-art generative compression methods at low ultra-low bitrates.

cs.CV↗

arXiv2025

Broken Lorentz symmetry and violation of the Wiedemann-Franz law in topological insulators

We study a two-dimensional topological insulator in the presence of a static non-uniform gravitational field, which mimics the variations in the temperature distribution. We derive an effective boundary free energy functional for the gravitational field and show that, in contrast to the case of massive Dirac fermions, the addition of a Newtonian mass term significantly modifies the quantum anomalous behavior of the system. A non-zero bulk thermal current appears, which violates the Wiedemann-Franz law. The systematic approach we develop to calculate the contribution of edge states to thermal Hall conductivity and energy magnetization can easily be extended to other models.

cond-mat.mes-hall↗

arXiv2025

DynamicLip: Shape-Independent Continuous Authentication via Lip Articulator Dynamics

Biometrics authentication has become increasingly popular due to its security and convenience; however, traditional biometrics are becoming less desirable in scenarios such as new mobile devices, Virtual Reality, and Smart Vehicles. For example, while face authentication is widely used, it suffers from significant privacy concerns. The collection of complete facial data makes it less desirable for privacy-sensitive applications. Lip authentication, on the other hand, has emerged as a promising biometrics method. However, existing lip-based authentication methods heavily depend on static lip shape when the mouth is closed, which can be less robust due to lip shape dynamic motion and can barely work when the user is speaking. In this paper, we revisit the nature of lip biometrics and extract shape-independent features from the lips. We study the dynamic characteristics of lip biometrics based on articulator motion. Building on the knowledge, we propose a system for shape-independent continuous authentication via lip articulator dynamics. This system enables robust, shape-independent and continuous authentication, making it particularly suitable for scenarios with high security and privacy requirements. We conducted comprehensive experiments in different environments and attack scenarios and collected a dataset of 50 subjects. The results indicate that our system achieves an overall accuracy of 99.06% and demonstrates robustness under advanced mimic attacks and AI deepfake attacks, making it a viable solution for continuous biometric authentication in various applications.

cs.CV↗

arXiv2024

Constructions of minimally $t$-tough regular graphs

A non-complete graph $G$ is said to be $t$-tough if for every vertex cut $S$ of $G$, the ratio of $|S|$ to the number of components of $G-S$ is at least $t$. The toughness $τ(G)$ of the graph $G$ is the maximum value of $t$ such that $G$ is $t$-tough. A graph $G$ is said to be minimally $t$-tough if $τ(G)=t$ and $τ(G-e)<t$ for every $e\in E(G)$. In 2003, Kriesell conjectured that every minimally $1$-tough graph contains a vertex of degree $2$. In 2018, Katona and Varga generalized this conjecture, asserting that every minimally $t$-tough graph contains a vertex of degree $\lceil 2t \rceil$. Recently, Zheng and Sun disproved the generalized Kriesell conjecture by constructing a family of $4$-regular graphs of even order. They also raised the question of whether there exist other minimally $t$-tough regular graphs that do not satisfy the generalized Kriesell conjecture. In this paper, we provide an affirmative answer by constructing a family of $4$-regular graphs of odd order, as well as a family of 6-regular graphs of order $3k+1~(k\geq 5)$.

math.CO↗

arXiv2024

Move-in-2D: 2D-Conditioned Human Motion Generation

Generating realistic human videos remains a challenging task, with the most effective methods currently relying on a human motion sequence as a control signal. Existing approaches often use existing motion extracted from other videos, which restricts applications to specific motion types and global scene matching. We propose Move-in-2D, a novel approach to generate human motion sequences conditioned on a scene image, allowing for diverse motion that adapts to different scenes. Our approach utilizes a diffusion model that accepts both a scene image and text prompt as inputs, producing a motion sequence tailored to the scene. To train this model, we collect a large-scale video dataset featuring single-human activities, annotating each video with the corresponding human motion as the target output. Experiments demonstrate that our method effectively predicts human motion that aligns with the scene image after projection. Furthermore, we show that the generated motion sequence improves human motion quality in video synthesis tasks.

cs.CV↗

arXiv2024

Privacy-Preserving Low-Rank Adaptation against Membership Inference Attacks for Latent Diffusion Models

Low-rank adaptation (LoRA) is an efficient strategy for adapting latent diffusion models (LDMs) on a private dataset to generate specific images by minimizing the adaptation loss. However, the LoRA-adapted LDMs are vulnerable to membership inference (MI) attacks that can judge whether a particular data point belongs to the private dataset, thus leading to the privacy leakage. To defend against MI attacks, we first propose a straightforward solution: Membership-Privacy-preserving LoRA (MP-LoRA). MP-LoRA is formulated as a min-max optimization problem where a proxy attack model is trained by maximizing its MI gain while the LDM is adapted by minimizing the sum of the adaptation loss and the MI gain of the proxy attack model. However, we empirically find that MP-LoRA has the issue of unstable optimization, and theoretically analyze that the potential reason is the unconstrained local smoothness, which impedes the privacy-preserving adaptation. To mitigate this issue, we further propose a Stable Membership-Privacy-preserving LoRA (SMP-LoRA) that adapts the LDM by minimizing the ratio of the adaptation loss to the MI gain. Besides, we theoretically prove that the local smoothness of SMP-LoRA can be constrained by the gradient norm, leading to improved convergence. Our experimental results corroborate that SMP-LoRA can indeed defend against MI attacks and generate high-quality images. Our Code is available at \url{https://github.com/WilliamLUO0/StablePrivateLoRA}.

cs.LG↗

arXiv2024

STEAM-EEG: Spatiotemporal EEG Analysis with Markov Transfer Fields and Attentive CNNs

Electroencephalogram (EEG) signals play a pivotal role in biomedical research and clinical applications, including epilepsy diagnosis, sleep disorder analysis, and brain-computer interfaces. However, the effective analysis and interpretation of these complex signals often present significant challenges. This paper presents a novel approach that integrates computer graphics techniques with biological signal pattern recognition, specifically using Markov Transfer Fields (MTFs) for EEG time series imaging. The proposed framework (STEAM-EEG) employs the capabilities of MTFs to capture the spatiotemporal dynamics of EEG signals, transforming them into visually informative images. These images are then rendered, visualised, and modelled using state-of-the-art computer graphics techniques, thereby facilitating enhanced data exploration, pattern recognition, and decision-making. The code could be accessed from GitHub.

cs.CV↗

arXiv2024

GAF-FusionNet: Multimodal ECG Analysis via Gramian Angular Fields and Split Attention

Electrocardiogram (ECG) analysis plays a crucial role in diagnosing cardiovascular diseases, but accurate interpretation of these complex signals remains challenging. This paper introduces a novel multimodal framework(GAF-FusionNet) for ECG classification that integrates time-series analysis with image-based representation using Gramian Angular Fields (GAF). Our approach employs a dual-layer cross-channel split attention module to adaptively fuse temporal and spatial features, enabling nuanced integration of complementary information. We evaluate GAF-FusionNet on three diverse ECG datasets: ECG200, ECG5000, and the MIT-BIH Arrhythmia Database. Results demonstrate significant improvements over state-of-the-art methods, with our model achieving 94.5\%, 96.9\%, and 99.6\% accuracy on the respective datasets. Our code will soon be available at https://github.com/Cross-Innovation-Lab/GAF-FusionNet.git.

cs.CV↗

arXiv2024

Semantic Retrieval at Walmart

In product search, the retrieval of candidate products before re-ranking is more critical and challenging than other search like web search, especially for tail queries, which have a complex and specific search intent. In this paper, we present a hybrid system for e-commerce search deployed at Walmart that combines traditional inverted index and embedding-based neural retrieval to better answer user tail queries. Our system significantly improved the relevance of the search engine, measured by both offline and online evaluations. The improvements were achieved through a combination of different approaches. We present a new technique to train the neural model at scale. and describe how the system was deployed in production with little impact on response time. We highlight multiple learnings and practical tricks that were used in the deployment of this system.

cs.IR↗

arXiv2024

Interplay between topology and correlations in the second moiré band of twisted bilayer MoTe2

Topological flat bands formed in two-dimensional lattice systems offer unique opportunity to study the fractional phases of matter in the absence of an external magnetic field. Celebrated examples include fractional quantum anomalous Hall (FQAH) effects and fractional topological insulators. Recently, FQAH effects have been experimentally realized in both the twisted bilayer MoTe2 (tMoTe2) system and the rhombohedral stacked multilayer graphene/hBN moiré systems. To date, experimental studies mainly focus on the first moiré flat band, except a very recent work that studied novel transport properties in higher moiré bands of a 2.1° tMoTe2 device. Here, we present the systematical transport study of approximately 3° tMoTe2 devices, especially for the second moiré band. At ν = -2 and -4, time-reversal-symmetric single and double quantum spin Hall states formed, consistent with the previous observation in 2.1° tMoTe2 device. On the other hand, we observed ferromagnetism in the second moiré band, and a Chern insulator state driven by out-of-plane magnetic fields at ν = -3. At ν = -2.2 to -2.7, finite temperature resistivity minimum with 1/T scaling at low temperatures, and large out-of-plane negative magnetoresistance have been observed. Applying out-of-plane electric field can induce quantum phase transitions at both integer and fractional filling factors. Our studies pave the way for realizing tunable topological states and other unexpected magnetic phases beyond the first moiré flat band based on twisted MoTe2 platform.

cond-mat.mes-hall↗

arXiv2024

Automatic Differentiation-based Full Waveform Inversion with Flexible Workflows

Full waveform inversion (FWI) is able to construct high-resolution subsurface models by iteratively minimizing discrepancies between observed and simulated seismic data. However, its implementation can be rather involved for complex wave equations, objective functions, or regularization. Recently, automatic differentiation (AD) has proven to be effective in simplifying solutions of various inverse problems, including FWI. In this study, we present an open-source AD-based FWI framework (ADFWI), which is designed to simplify the design, development, and evaluation of novel approaches in FWI with flexibility. The AD-based framework not only includes forword modeling and associated gradient computations for wave equations in various types of media from isotropic acoustic to vertically or horizontally transverse isotropic elastic, but also incorporates a suite of objective functions, regularization techniques, and optimization algorithms. By leveraging state-of-the-art AD, objective functions such as soft dynamic time warping and Wasserstein distance, which are difficult to apply in traditional FWI are also easily integrated into ADFWI. In addition, ADFWI is integrated with deep learning for implicit model reparameterization via neural networks, which not only introduces learned regularization but also allows rapid estimation of uncertainty through dropout. To manage high memory demands in large-scale inversion associated with AD, the proposed framework adopts strategies such as mini-batch and checkpointing. Through comprehensive evaluations, we demonstrate the novelty, practicality and robustness of ADFWI, which can be used to address challenges in FWI and as a workbench for prompt experiments and the development of new inversion strategies.

cs.LG↗

arXiv2024

Problem-Driven Scenario Reduction Framework for Power System Stochastic Operation

Scenario reduction (SR) aims to identify a small yet representative scenario set to depict the underlying uncertainty, which is critical to scenario-based stochastic optimization (SBSO) of power systems. Existing SR techniques commonly aim to achieve statistical approximation to the original scenario set. However, SR and SBSO are commonly considered as two distinct and decoupled processes, which cannot guarantee a superior approximation of the original optimality. Instead, this paper incorporates the SBSO problem structure into the SR process and introduces a novel problem-driven scenario reduction (PDSR) framework. Specifically, we project the original scenario set in distribution space onto the mutual decision applicability between scenarios in problem space. Subsequently, the SR process, embedded by a distinctive problem-driven distance metric, is rendered as a mixed-integer linear programming formulation to obtain the representative scenario set while minimizing the optimality gap. Furthermore, ex-ante and ex-post problem-driven evaluation indices are proposed to evaluate the SR performance. Numerical experiments on two two-stage stochastic economic dispatch problems validate the effectiveness of PDSR, and demonstrate that PDSR significantly outperforms existing SR methods by identifying salient (e.g., worst-case) scenarios, and achieving an optimality gap of less than 0.1% within acceptable computation time.

math.OC↗

arXiv2024

EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation

Video generation has emerged as a promising tool for world simulation, leveraging visual data to replicate real-world environments. Within this context, egocentric video generation, which centers on the human perspective, holds significant potential for enhancing applications in virtual reality, augmented reality, and gaming. However, the generation of egocentric videos presents substantial challenges due to the dynamic nature of egocentric viewpoints, the intricate diversity of actions, and the complex variety of scenes encountered. Existing datasets are inadequate for addressing these challenges effectively. To bridge this gap, we present EgoVid-5M, the first high-quality dataset specifically curated for egocentric video generation. EgoVid-5M encompasses 5 million egocentric video clips and is enriched with detailed action annotations, including fine-grained kinematic control and high-level textual descriptions. To ensure the integrity and usability of the dataset, we implement a sophisticated data cleaning pipeline designed to maintain frame consistency, action coherence, and motion smoothness under egocentric conditions. Furthermore, we introduce EgoDreamer, which is capable of generating egocentric videos driven simultaneously by action descriptions and kinematic control signals. The EgoVid-5M dataset, associated action annotations, and all data cleansing metadata will be released for the advancement of research in egocentric video generation.

cs.CV↗

arXiv2024

World Models: The Safety Perspective

With the proliferation of the Large Language Model (LLM), the concept of World Models (WM) has recently attracted a great deal of attention in the AI research community, especially in the context of AI agents. It is arguably evolving into an essential foundation for building AI agent systems. A WM is intended to help the agent predict the future evolution of environmental states or help the agent fill in missing information so that it can plan its actions and behave safely. The safety property of WM plays a key role in their effective use in critical applications. In this work, we review and analyze the impacts of the current state-of-the-art in WM technology from the point of view of trustworthiness and safety based on a comprehensive survey and the fields of application envisaged. We provide an in-depth analysis of state-of-the-art WMs and derive technical research challenges and their impact in order to call on the research community to collaborate on improving the safety and trustworthiness of WM.

cs.AI↗

arXiv2024

Adaptive Conditional Expert Selection Network for Multi-domain Recommendation

Mixture-of-Experts (MOE) has recently become the de facto standard in Multi-domain recommendation (MDR) due to its powerful expressive ability. However, such MOE-based method typically employs all experts for each instance, leading to scalability issue and low-discriminability between domains and experts. Furthermore, the design of commonly used domain-specific networks exacerbates the scalability issues. To tackle the problems, We propose a novel method named CESAA consists of Conditional Expert Selection (CES) Module and Adaptive Expert Aggregation (AEA) Module to tackle these challenges. Specifically, CES first combines a sparse gating strategy with domain-shared experts. Then AEA utilizes mutual information loss to strengthen the correlations between experts and specific domains, and significantly improve the distinction between experts. As a result, only domain-shared experts and selected domain-specific experts are activated for each instance, striking a balance between computational efficiency and model performance. Experimental results on both public ranking and industrial retrieval datasets verify the effectiveness of our method in MDR tasks.

cs.LG↗

arXiv2024

'No' Matters: Out-of-Distribution Detection in Multimodality Long Dialogue

Out-of-distribution (OOD) detection in multimodal contexts is essential for identifying deviations in combined inputs from different modalities, particularly in applications like open-domain dialogue systems or real-life dialogue interactions. This paper aims to improve the user experience that involves multi-round long dialogues by efficiently detecting OOD dialogues and images. We introduce a novel scoring framework named Dialogue Image Aligning and Enhancing Framework (DIAEF) that integrates the visual language models with the novel proposed scores that detect OOD in two key scenarios (1) mismatches between the dialogue and image input pair and (2) input pairs with previously unseen labels. Our experimental results, derived from various benchmarks, demonstrate that integrating image and multi-round dialogue OOD detection is more effective with previously unseen labels than using either modality independently. In the presence of mismatched pairs, our proposed score effectively identifies these mismatches and demonstrates strong robustness in long dialogues. This approach enhances domain-aware, adaptive conversational agents and establishes baselines for future studies.

cs.CL↗

arXiv2024

Bayesian-guided Label Mapping for Visual Reprogramming

Visual reprogramming (VR) leverages the intrinsic capabilities of pretrained vision models by adapting their input or output interfaces to solve downstream tasks whose labels (i.e., downstream labels) might be totally different from the labels associated with the pretrained models (i.e., pretrained labels). When adapting the output interface, label mapping methods transform the pretrained labels to downstream labels by establishing a gradient-free one-to-one correspondence between the two sets of labels. However, in this paper, we reveal that one-to-one mappings may overlook the complex relationship between pretrained and downstream labels. Motivated by this observation, we propose a Bayesian-guided Label Mapping (BLM) method. BLM constructs an iteratively-updated probabilistic label mapping matrix, with each element quantifying a pairwise relationship between pretrained and downstream labels. The assignment of values to the constructed matrix is guided by Bayesian conditional probability, considering the joint distribution of the downstream labels and the labels predicted by the pretrained model on downstream samples. Experiments conducted on both pretrained vision models (e.g., ResNeXt) and vision-language models (e.g., CLIP) demonstrate the superior performance of BLM over existing label mapping methods. The success of BLM also offers a probabilistic lens through which to understand and analyze the effectiveness of VR. Our code is available at https://github.com/tmlr-group/BayesianLM.

cs.LG↗