arXiv subjects

Xiangwen Deng

Publications and source records attributed to Xiangwen Deng.

8 recordsLinked to original sources

arXiv2026

Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

Knowledge-based visual question answering (KB-VQA) requires vision-language models to understand images and use external knowledge, especially for rare entities and long-tail facts. Most existing retrieval-augmented generation (RAG) methods adopt a fixed pipeline that sequentially retrieves information, filters it, and then produces an answer. Such a design makes it difficult to adapt to diverse question types. Moreover, it separates retrieval from reasoning, making it hard for the model to decide when to search, how to refine queries, or when to stop. As a result, the retrieved evidence is often poorly aligned with the question. To address these limitations, we reformulate KB-VQA as a search-agent problem and model the solving process as a multi-step decision-making procedure. At each step, the agent selects one of four actions-Answer, Image Retrieval, Text Retrieval, and Caption-based on its current information state. We further design an automated pipeline to collect multi-step trajectories that record the agent's reasoning process, tool usage, and intermediate decisions. These trajectories are then used as supervision for fine-tuning. Experiments on InfoSeek and E-VQA demonstrate that our method achieves state-of-the-art performance, consistently outperforming prior baselines and confirming the effectiveness of our framework.

cs.CV↗

arXiv2026

Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering

Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.

cs.CV↗

arXiv2026

Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSampling

In the unsupervised self-evolution of Multimodal Large Language Models, the quality of feedback signals during post-training is pivotal for stable and effective learning. However, existing self-evolution methods predominantly rely on majority voting to select the most frequent output as the pseudo-golden answer, which may stem from the model's intrinsic biases rather than guaranteeing the objective correctness of the reasoning paths. To counteract the degradation, we propose Continuous Softened Retracing reSampling (CSRS) in MLLM self-evolution. Specifically, we introduce a Retracing Re-inference Mechanism (RRM) that the model re-inferences from anchor points to expand the exploration of long-tail reasoning paths. Simultaneously, we propose Softened Frequency Reward (SFR), which replaces binary rewards with continuous signals, calibrating reward based on the answers' frequency across sampled reasoning sets. Furthermore, incorporated with Visual Semantic Perturbation (VSP), CSRS ensures the model prioritizes mathematical logic over visual superficiality. Experimental results demonstrate that CSRS significantly enhances the reasoning performance of Qwen2.5-VL-7B on benchmarks such as MathVision. We achieve state-of-the-art (SOTA) results in unsupervised self-evolution on geometric tasks. Our code is avaible at https://github.com/yyy195/CSRS.

cs.CV↗

arXiv2023

ITportrait: Image-Text Coupled 3D Portrait Domain Adaptation

Domain adaptation of 3D portraits has gained more and more attention. However, the transfer mechanism of existing methods is mainly based on vision or language, which ignores the potential of vision-language combined guidance. In this paper, we propose an Image-Text multi-modal framework, namely Image and Text portrait (ITportrait), for 3D portrait domain adaptation. ITportrait relies on a two-stage alternating training strategy. In the first stage, we employ a 3D Artistic Paired Transfer (APT) method for image-guided style transfer. APT constructs paired photo-realistic portraits to obtain accurate artistic poses, which helps ITportrait to achieve high-quality 3D style transfer. In the second stage, we propose a 3D Image-Text Embedding (ITE) approach in the CLIP space. ITE uses a threshold function to self-adaptively control the optimization direction of images or texts in the CLIP space. Comprehensive experiments prove that our ITportrait achieves state-of-the-art (SOTA) results and benefits downstream tasks. All source codes and pre-trained models will be released to the public.

cs.MM↗

arXiv2023

3D Face Arbitrary Style Transfer

Style transfer of 3D faces has gained more and more attention. However, previous methods mainly use images of artistic faces for style transfer while ignoring arbitrary style images such as abstract paintings. To solve this problem, we propose a novel method, namely Face-guided Dual Style Transfer (FDST). To begin with, FDST employs a 3D decoupling module to separate facial geometry and texture. Then we propose a style fusion strategy for facial geometry. Subsequently, we design an optimization-based DDSG mechanism for textures that can guide the style transfer by two style images. Besides the normal style image input, DDSG can utilize the original face input as another style input as the face prior. By this means, high-quality face arbitrary style transfer results can be obtained. Furthermore, FDST can be applied in many downstream tasks, including region-controllable style transfer, high-fidelity face texture reconstruction, large-pose face reconstruction, and artistic face reconstruction. Comprehensive quantitative and qualitative results show that our method can achieve comparable performance. All source codes and pre-trained weights will be released to the public.

cs.CV↗

arXiv2022

EEG-based Emotion Recognition with Spatial and Functional Brain Mapping of CNS and PNS Signals

Emotion plays a significant role in our daily life. Recognition of emotion is wide-spread in the field of health care and human-computer interaction. Emotion is the result of the coordinated activities of cortical and subcortical neural processes, which correlate to specific physiological responses. However, the existing emotion recognition techniques failed to combine various physiological signals as one integrated feature representation. Meanwhile, many researchers ignored the problem of over-fitting model with high accuracy, which was actually false high accuracy caused by improper pre-processing. In this paper, sigmoid baseline filtering is conducted to solve the over-fitting problem from source. To construct a physiological-based algorithm, a 3D spatial and functional brain mapping is proposed based on human physiological mechanism and international electrode system, which combines the signals of the central and peripheral nervous system together. By combining the baseline filtering, 3D brain mapping, and simple 4D-CNN, a novel emotion recognition model is finally proposed. Experiment results demonstrate that the performance of the proposed model is comparable to the state of art algorithms.

cs.MM↗

arXiv2021

SFE-Net: EEG-based Emotion Recognition with Symmetrical Spatial Feature Extraction

Emotion recognition based on EEG (electroencephalography) has been widely used in human-computer interaction, distance education and health care. However, the conventional methods ignore the adjacent and symmetrical characteristics of EEG signals, which also contain salient information related to emotion. In this paper, a spatial folding ensemble network (SFE-Net) is presented for EEG feature extraction and emotion recognition. Firstly, for the undetected area between EEG electrodes, an improved Bicubic-EEG interpolation algorithm is developed for EEG channels information completion, which allows us to extract a wider range of adjacent space features. Then, motivated by the spatial symmetric mechanism of human brain, we fold the input EEG channels data with five different symmetrical strategies, which enable the proposed network to extract the information of space features of EEG signals more effectively. Finally, a 3DCNN-based spatial, temporal extraction, and a multi-voting strategy of ensemble learning are integrated to model a new neural network. With this network, the spatial features of different symmetric folding signals can be extracted simultaneously, which greatly improves the robustness and accuracy of emotion recognition. The experimental results on DEAP and SEED datasets show that the proposed algorithm has comparable performance in terms of recognition accuracy.

eess.SP↗

arXiv2020

Portable and wireless signal transducer for field testing of environmental sensors based on 2D materials

In this paper we present the design and fabrication of a portable device for environmental monitoring applications. This novel hand-held apparatus monitors the changes in the resistance of a sensing surface with a high accuracy and resolution and transmits the recorded data wirelessly to a cellphone. Such a design offers a solution for field testing of environmental sensors. The tested sensing surface in this study is based on an ultrathin material: graphene, which is placed on the surface of a Si/SiO2 wafer. This signal transducer and wireless communication system form together an ideal platform to harvest the sensitivity and selectivity of 2D materials for gas sensing applications.

physics.ins-det↗