arXiv subjects

Weidong Geng

Publications and source records attributed to Weidong Geng.

10 recordsLinked to original sources

arXiv2026

FT-RAG: A Fine-grained Retrieval-Augmented Generation Framework for Complex Table Reasoning

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding responses in external knowledge during inference. However, conventiona RAG systems under-perform on structured tabular data, largely due to coarse retrieval granularity and insufficient table semantic comprehension. To address these limitations, we introduce FT-RAG, a fine-grained framework that employs knowledge association by decomposing tables into entry-level semantic units to construct a structured graph. FT-RAG employs a structural neighbor expansion mechanism to find semantically connected entities during graph retrieval, followed by multi-modal fusion to consolidate the context of table retrieval results. Further, to address the scarcity of specialized datasets in this domain, we introduce Multi-Table-RAG-Lib, a benchmark comprising 9870 QA pairs with high complexity and difficulty, curated to demand multi-table integration and text-table information fusion for reasoning. FT-RAG surpasses top-performing baselines across all metrics, achieving a 23.5\% and 59.2\% improvement in table-level and cell-level Hit Rates, respectively. Generation performance also sees a remarkable 62.2\% increase in exact value accuracy recall. These metrics verify the framework's effectiveness in factual grounding across both pure tabular and heterogeneous table-text contexts. Therefore, our method establishes a new state-of-the-art performance for complex reasoning over mixed-modality documents.

cs.CL↗

arXiv2024

DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2

Speech-driven gesture generation using transformer-based generative models represents a rapidly advancing area within virtual human creation. However, existing models face significant challenges due to their quadratic time and space complexities, limiting scalability and efficiency. To address these limitations, we introduce DiM-Gestor, an innovative end-to-end generative model leveraging the Mamba-2 architecture. DiM-Gestor features a dual-component framework: (1) a fuzzy feature extractor and (2) a speech-to-gesture mapping module, both built on the Mamba-2. The fuzzy feature extractor, integrated with a Chinese Pre-trained Model and Mamba-2, autonomously extracts implicit, continuous speech features. These features are synthesized into a unified latent representation and then processed by the speech-to-gesture mapping module. This module employs an Adaptive Layer Normalization (AdaLN)-enhanced Mamba-2 mechanism to uniformly apply transformations across all sequence tokens. This enables precise modeling of the nuanced interplay between speech features and gesture dynamics. We utilize a diffusion model to train and infer diverse gesture outputs. Extensive subjective and objective evaluations conducted on the newly released Chinese Co-Speech Gestures dataset corroborate the efficacy of our proposed model. Compared with Transformer-based architecture, the assessments reveal that our approach delivers competitive results and significantly reduces memory usage, approximately 2.4 times, and enhances inference speeds by 2 to 4 times. Additionally, we released the CCG dataset, a Chinese Co-Speech Gestures dataset, comprising 15.97 hours (six styles across five scenarios) of 3D full-body skeleton gesture motion performed by professional Chinese TV broadcasters.

cs.SD↗

arXiv2024

POS: A Prompts Optimization Suite for Augmenting Text-to-Video Generation

This paper targets to enhance the diffusion-based text-to-video generation by improving the two input prompts, including the noise and the text. Accommodated with this goal, we propose POS, a training-free Prompt Optimization Suite to boost text-to-video models. POS is motivated by two observations: (1) Video generation shows instability in terms of noise. Given the same text, different noises lead to videos that differ significantly in terms of both frame quality and temporal consistency. This observation implies that there exists an optimal noise matched to each textual input; To capture the potential noise, we propose an optimal noise approximator to approach the potential optimal noise. Particularly, the optimal noise approximator initially searches a video that closely relates to the text prompt and then inverts it into the noise space to serve as an improved noise prompt for the textual input. (2) Improving the text prompt via LLMs often causes semantic deviation. Many existing text-to-vision works have utilized LLMs to improve the text prompts for generation enhancement. However, existing methods often neglect the semantic alignment between the original text and the rewritten one. In response to this issue, we design a semantic-preserving rewriter to impose contraints in both rewritng and denoising phrases to preserve the semantic consistency. Extensive experiments on popular benchmarks show that our POS can improve the text-to-video models with a clear margin. The code will be open-sourced.

cs.CV↗

arXiv2024

MCM: Multi-condition Motion Synthesis Framework

Conditional human motion synthesis (HMS) aims to generate human motion sequences that conform to specific conditions. Text and audio represent the two predominant modalities employed as HMS control conditions. While existing research has primarily focused on single conditions, the multi-condition human motion synthesis remains underexplored. In this study, we propose a multi-condition HMS framework, termed MCM, based on a dual-branch structure composed of a main branch and a control branch. This framework effectively extends the applicability of the diffusion model, which is initially predicated solely on textual conditions, to auditory conditions. This extension encompasses both music-to-dance and co-speech HMS while preserving the intrinsic quality of motion and the capabilities for semantic association inherent in the original model. Furthermore, we propose the implementation of a Transformer-based diffusion model, designated as MWNet, as the main branch. This model adeptly apprehends the spatial intricacies and inter-joint correlations inherent in motion sequences, facilitated by the integration of multi-wise self-attention modules. Extensive experiments show that our method achieves competitive results in single-condition and multi-condition HMS tasks.

cs.CV↗

arXiv2024

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Speech-driven gesture generation is an emerging field within virtual human creation. However, a significant challenge lies in accurately determining and processing the multitude of input features (such as acoustic, semantic, emotional, personality, and even subtle unknown features). Traditional approaches, reliant on various explicit feature inputs and complex multimodal processing, constrain the expressiveness of resulting gestures and limit their applicability. To address these challenges, we present Persona-Gestor, a novel end-to-end generative model designed to generate highly personalized 3D full-body gestures solely relying on raw speech audio. The model combines a fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture. The fuzzy feature extractor harnesses a fuzzy inference strategy that automatically infers implicit, continuous fuzzy features. These fuzzy features, represented as a unified latent feature, are fed into the AdaLN transformer. The AdaLN transformer introduces a conditional mechanism that applies a uniform function across all tokens, thereby effectively modeling the correlation between the fuzzy features and the gesture sequence. This module ensures a high level of gesture-speech synchronization while preserving naturalness. Finally, we employ the diffusion model to train and infer various gestures. Extensive subjective and objective evaluations on the Trinity, ZEGGS, and BEAT datasets confirm our model's superior performance to the current state-of-the-art approaches. Persona-Gestor improves the system's usability and generalization capabilities, setting a new benchmark in speech-driven gesture synthesis and broadening the horizon for virtual human technology. Supplementary videos and code can be accessed at https://zf223669.github.io/Diffmotion-v2-website/

cs.SD↗

arXiv2023

MCM: Multi-condition Motion Synthesis Framework for Multi-scenario

The objective of the multi-condition human motion synthesis task is to incorporate diverse conditional inputs, encompassing various forms like text, music, speech, and more. This endows the task with the capability to adapt across multiple scenarios, ranging from text-to-motion and music-to-dance, among others. While existing research has primarily focused on single conditions, the multi-condition human motion generation remains underexplored. In this paper, we address these challenges by introducing MCM, a novel paradigm for motion synthesis that spans multiple scenarios under diverse conditions. The MCM framework is able to integrate with any DDPM-like diffusion model to accommodate multi-conditional information input while preserving its generative capabilities. Specifically, MCM employs two-branch architecture consisting of a main branch and a control branch. The control branch shares the same structure as the main branch and is initialized with the parameters of the main branch, effectively maintaining the generation ability of the main branch and supporting multi-condition input. We also introduce a Transformer-based diffusion model MWNet (DDPM-like) as our main branch that can capture the spatial complexity and inter-joint correlations in motion sequences through a channel-dimension self-attention module. Quantitative comparisons demonstrate that our approach achieves SoTA results in both text-to-motion and competitive results in music-to-dance tasks, comparable to task-specific methods. Furthermore, the qualitative evaluation shows that MCM not only streamlines the adaptation of methodologies originally designed for text-to-motion tasks to domains like music-to-dance and speech-to-gesture, eliminating the need for extensive network re-configurations but also enables effective multi-condition modal control, realizing "once trained is motion need".

cs.CV↗

arXiv2018

Ultrasensitive hybrid optical skin

Electronic skin, a class of wearable electronic sensors that mimic the functionalities of human skin, has made remarkable success in applications including health monitoring, human-machine interaction and electronic-biological interfaces. While electronic skin continues to achieve higher sensitivity and faster response, its ultimate performance is fundamentally limited by the nature of low-frequency AC currents in electronic circuitries. Here we demonstrate highly sensitive optical skin (O-skin) in which the primary sensory elements are optically driven. The simple construction of the sensors is achieved by embedding glass micro/nanofibers (MNFs) in thin layers of polydimethylsiloxane (PDMS). Enabled by the highly sensitive power-leakage response of the guided modes from the MNF upon external stimuli, our optical sensors show ultrahigh sensitivity (1870/kPa), low detection limit (7 mPa) and fast response (10 microseconds) for pressure sensing, significantly exceeding the performance metrics of state-of-the-art electronic skins. Electromagnetic interference (EMI)-free detection of high-frequency vibrations, wrist pulse and human voice are realized. Moreover, a five-sensor optical data glove and a 2x2-MNF tactile sensor are demonstrated. Our results pave the way toward wearable optical devices ranging from ultrasensitive flexible sensors to optical skins.

physics.app-ph↗

arXiv2016

Learning to Sketch Human Facial Portraits using Personal Styles by Case-Based Reasoning

This paper employs case-based reasoning (CBR) to capture the personal styles of individual artists and generate the human facial portraits from photos accordingly. For each human artist to be mimicked, a series of cases are firstly built-up from her/his exemplars of source facial photo and hand-drawn sketch, and then its stylization for facial photo is transformed as a style-transferring process of iterative refinement by looking-for and applying best-fit cases in a sense of style optimization. Two models, fitness evaluation model and parameter estimation model, are learned for case retrieval and adaptation respectively from these cases. The fitness evaluation model is to decide which case is best-fitted to the sketching of current interest, and the parameter estimation model is to automate case adaptation. The resultant sketch is synthesized progressively with an iterative loop of retrieval and adaptation of candidate cases until the desired aesthetic style is achieved. To explore the effectiveness and advantages of the novel approach, we experimentally compare the sketch portraits generated by the proposed method with that of a state-of-the-art example-based facial sketch generation algorithm as well as a couple commercial software packages. The comparisons reveal that our CBR based synthesis method for facial portraits is superior both in capturing and reproducing artists' personal illustration styles to the peer methods.

cs.CV↗

arXiv2015

Quantum Interference Induced Photon Blockade in a Coupled Single Quantum Dot-Cavity System

We propose an experimental scheme to implement a strong photon blockade with a single quantum dot coupled to a nanocavity. The photon blockade effect can be tremendously enhanced by driving the cavity and the quantum dot simultaneously with two classical laser fields. This enhancement of photon blockade is ascribed to the quantum interference effect to avoid two-photon excitation of the cavity field. Comparing with Jaynes-Cummings model, the second-order correlation function at zero time delay $g^{(2)}(0)$ in our scheme can be reduced by two orders of magnitude and the system sustains a large intracavity photon number. A red (blue) cavity-light detuning asymmetry for photon quantum statistics with bunching or antibunching characteristics is also observed. The photon blockade effect has a controllable flexibility by tuning the relative phase between the two pumping laser fields and the Rabi coupling strength between the quantum dot and the pumping field. Moreover, the photon blockade scheme based on quantum interference mechanism does not require a strong coupling strength between the cavity and the quantum dot, even with the pure dephasing of the system. This simple proposal provides an effective way for potential applications in solid state quantum computation and quantum information processing.

quant-ph↗

arXiv2014

Charge state control in single InAs/GaAs quantum dots by external electric and magnetic fields

We report a photoluminescence (PL) spectroscopy study of charge state control in single self-assembled InAs/GaAs quantum dots by applying electric and/or magnetic fields at 4.2 K. Neutral and charged exciton complexes were observed under applied bias voltages from -0.5 V to 0.5 V by controlling the carrier tunneling. The highly negatively charged exciton emission becomes stronger with increasing pumping power, arising from the fact that electrons have a smaller effective mass than holes and are more easily captured by the quantum dots. The integrated PL intensity of negatively charged excitons is affected significantly by a magnetic field applied along the sample growth axis. This observation is explained by a reduction in the electron drift velocity caused by an applied magnetic field, which increases the probability of non-resonantly excited electrons being trapped by localized potentials at the wetting layer interface, and results in fewer electrons distributed in the quantum dots. The hole drift velocity is also affected by the magnetic field, but it is much weaker.

cond-mat.mes-hall↗