arXiv subjects

Xin Deng

Publications and source records attributed to Xin Deng.

At least 37 records · Page 2Linked to original sources

arXiv2024

Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback

Many approaches to Natural Language Processing (NLP) tasks often treat them as single-step problems, where an agent receives an instruction, executes it, and is evaluated based on the final outcome. However, human language is inherently interactive, as evidenced by the back-and-forth nature of human conversations. In light of this, we posit that human-AI collaboration should also be interactive, with humans monitoring the work of AI agents and providing feedback that the agent can understand and utilize. Further, the AI agent should be able to detect when it needs additional information and proactively ask for help. Enabling this scenario would lead to more natural, efficient, and engaging human-AI collaborations. In this work, we explore these directions using the challenging task defined by the IGLU competition, an interactive grounded language understanding task in a MineCraft-like world. We explore multiple types of help players can give to the AI to guide it and analyze the impact of this help in AI behavior, resulting in performance improvements.

cs.CL↗

arXiv2023

Making Large Language Models Interactive: A Pioneer Study on Supporting Complex Information-Seeking Tasks with Implicit Constraints

Current interactive systems with natural language interfaces lack the ability to understand a complex information-seeking request which expresses several implicit constraints at once, and there is no prior information about user preferences e.g.,"find hiking trails around San Francisco which are accessible with toddlers and have beautiful scenery in summer", where output is a list of possible suggestions for users to start their exploration. In such scenarios, user requests can be issued in one shot in the form of a complex and long query, unlike conversational and exploratory search models, where require short utterances or queries are often presented to the system step by step. We have designed and deployed a platform to collect the data from approaching such complex interactive systems. Moreover, despite with the current advancement of generative language models these models suffer from hallucination in providing accurate factual knowledge. All language models are mostly trained in large part on web-scraped data from the past, which usually is not useful for immediate users' needs. In this article, we propose an IA that leverages Large Language Models (LLM) for complex request understanding and makes it interactive using Reinforcement learning that allows intricately refine user requests by making them complete, leading to better retrieval and reduce LLMs hallucination problems for current user needs. To demonstrate the performance of the proposed modeling paradigm, we have adopted various pre-retrieval metrics that capture the extent to which guided interactions with our system yield better retrieval results. Through extensive experimentation, we demonstrated that our method significantly outperforms several robust baselines.

cs.IR↗

arXiv2023

Pixel Adapter: A Graph-Based Post-Processing Approach for Scene Text Image Super-Resolution

Current Scene text image super-resolution approaches primarily focus on extracting robust features, acquiring text information, and complex training strategies to generate super-resolution images. However, the upsampling module, which is crucial in the process of converting low-resolution images to high-resolution ones, has received little attention in existing works. To address this issue, we propose the Pixel Adapter Module (PAM) based on graph attention to address pixel distortion caused by upsampling. The PAM effectively captures local structural information by allowing each pixel to interact with its neighbors and update features. Unlike previous graph attention mechanisms, our approach achieves 2-3 orders of magnitude improvement in efficiency and memory utilization by eliminating the dependency on sparse adjacency matrices and introducing a sliding window approach for efficient parallel computation. Additionally, we introduce the MLP-based Sequential Residual Block (MSRB) for robust feature extraction from text images, and a Local Contour Awareness loss ($\mathcal{L}_{lca}$) to enhance the model's perception of details. Comprehensive experiments on TextZoom demonstrate that our proposed method generates high-quality super-resolution images, surpassing existing methods in recognition accuracy. For single-stage and multi-stage strategies, we achieved improvements of 0.7\% and 2.6\%, respectively, increasing the performance from 52.6\% and 53.7\% to 53.3\% and 56.3\%. The code is available at https://github.com/wenyu1009/RTSRN.

cs.CV↗

arXiv2023

Uncertainty Guided Adaptive Warping for Robust and Efficient Stereo Matching

Correlation based stereo matching has achieved outstanding performance, which pursues cost volume between two feature maps. Unfortunately, current methods with a fixed model do not work uniformly well across various datasets, greatly limiting their real-world applicability. To tackle this issue, this paper proposes a new perspective to dynamically calculate correlation for robust stereo matching. A novel Uncertainty Guided Adaptive Correlation (UGAC) module is introduced to robustly adapt the same model for different scenarios. Specifically, a variance-based uncertainty estimation is employed to adaptively adjust the sampling area during warping operation. Additionally, we improve the traditional non-parametric warping with learnable parameters, such that the position-specific weights can be learned. We show that by empowering the recurrent network with the UGAC module, stereo matching can be exploited more robustly and effectively. Extensive experiments demonstrate that our method achieves state-of-the-art performance over the ETH3D, KITTI, and Middlebury datasets when employing the same fixed model over these datasets without any retraining procedure. To target real-time applications, we further design a lightweight model based on UGAC, which also outperforms other methods over KITTI benchmarks with only 0.6 M parameters.

cs.CV↗

arXiv2023

PointVector: A Vector Representation In Point Cloud Analysis

In point cloud analysis, point-based methods have rapidly developed in recent years. These methods have recently focused on concise MLP structures, such as PointNeXt, which have demonstrated competitiveness with Convolutional and Transformer structures. However, standard MLPs are limited in their ability to extract local features effectively. To address this limitation, we propose a Vector-oriented Point Set Abstraction that can aggregate neighboring features through higher-dimensional vectors. To facilitate network optimization, we construct a transformation from scalar to vector using independent angles based on 3D vector rotations. Finally, we develop a PointVector model that follows the structure of PointNeXt. Our experimental results demonstrate that PointVector achieves state-of-the-art performance $\textbf{72.3\% mIOU}$ on the S3DIS Area 5 and $\textbf{78.4\% mIOU}$ on the S3DIS (6-fold cross-validation) with only $\textbf{58\%}$ model parameters of PointNeXt. We hope our work will help the exploration of concise and effective feature representations. The code will be released soon.

cs.CV↗

arXiv2023

DAQE: Enhancing the Quality of Compressed Images by Exploiting the Inherent Characteristic of Defocus

Image defocus is inherent in the physics of image formation caused by the optical aberration of lenses, providing plentiful information on image quality. Unfortunately, existing quality enhancement approaches for compressed images neglect the inherent characteristic of defocus, resulting in inferior performance. This paper finds that in compressed images, significantly defocused regions have better compression quality, and two regions with different defocus values possess diverse texture patterns. These observations motivate our defocus-aware quality enhancement (DAQE) approach. Specifically, we propose a novel dynamic region-based deep learning architecture of the DAQE approach, which considers the regionwise defocus difference of compressed images in two aspects. (1) The DAQE approach employs fewer computational resources to enhance the quality of significantly defocused regions and more resources to enhance the quality of other regions; (2) The DAQE approach learns to separately enhance diverse texture patterns for regions with different defocus values, such that texture-specific enhancement can be achieved. Extensive experiments validate the superiority of our DAQE approach over state-of-the-art approaches in terms of quality enhancement and resource savings.

eess.IV↗

arXiv2022

Elastic anisotropy of lizardite at subduction zone conditions

Subduction zones transport water into Earth's deep interior through slab subduction. Serpentine minerals, the primary hydration product of ultramafic peridotite, are abundant in most subduction zones. Characterization of their high-temperature elasticity, particularly their anisotropy, will help us better estimate the extent of mantle serpentinization and the Earth's deep water cycle. Lizardite, the low-temperature polymorph of serpentine, is stable under the P-T conditions of cold subduction slabs (< 260°C at 2 GPa), and its high-temperature elasticity remains unknown. Here we report ab initio elasticity and acoustic wave velocities of lizardite at P-T conditions of subduction zones. Our static results agree with previous studies. Its high-temperature velocities are much higher than previous experimental-based lizardite estimates with chrysotile but closer to antigorite velocities. The elastic anisotropy of lizardite is much larger than that of antigorite and could better account for the observed large shear-wave splitting in some cold slabs such as Tonga.

physics.geo-ph↗

arXiv2022

Compression-Based Optimizations for Out-of-Core GPU Stencil Computation

An out-of-core stencil computation code handles large data whose size is beyond the capacity of GPU memory. Whereas, such an code requires streaming data to and from the GPU frequently. As a result, data movement between the CPU and GPU usually limits the performance. In this work, compression-based optimizations are proposed. First, an on-the-fly compression technique is applied to an out-of-core stencil code, reducing the CPU-GPU memory copy. Secondly, a single working buffer technique is used to reduce GPU memory consumption. Experimental results show that the stencil code using the proposed techniques achieved 1.1x speed and reduced GPU memory consumption by 33.0\% on an NVIDIA Tesla V100 GPU.

cs.DC↗

arXiv2021

Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos

Visual and audio events simultaneously occur and both attract attention. However, most existing saliency prediction works ignore the influence of audio and only consider vision modality. In this paper, we propose a multitask learning method for visual-audio saliency prediction and sound source localization on multi-face video by leveraging visual, audio and face information. Specifically, we first introduce a large-scale database of multi-face video in visual-audio condition (MVVA), containing eye-tracking data and sound source annotations. Using this database, we find that sound influences human attention, and conversly attention offers a cue to determine sound source on multi-face video. Guided by these findings, a visual-audio multi-task network (VAM-Net) is introduced to predict saliency and locate sound source. VAM-Net consists of three branches corresponding to visual, audio and face modalities. Visual branch has a two-stream architecture to capture spatial and temporal information. Face and audio branches encode audio signals and faces, respectively. Finally, a spatio-temporal multi-modal graph (STMG) is constructed to model the interaction among multiple faces. With joint optimization of these branches, the intrinsic correlation of the tasks of saliency prediction and sound source localization is utilized and their performance is boosted by each other. Experiments show that the proposed method outperforms 12 state-of-the-art saliency prediction methods, and achieves competitive results in sound source localization.

cs.CV↗

arXiv2021

Partially Diffusive Helium-Silica Compound in the Deep Interiors of Giant Planets

Helium is the second most abundant element in the universe, and together with silica, they are major components of giant planets. Exploring the reactivity and state of helium and silica under high pressure is of fundamental importance for developing and understanding of the evolution and internal structure of giant planets. Here, using first-principles calculations and crystal structure predictions, we identify four stable phases of a helium-silica compound with seven/eight-coordinated silicon atoms at pressure range of 600-4000 GPa, corresponding to the interior condition of the outer planets in the solar system. The density of HeSiO2 agrees with current structure models of the planets. This helium-silica compound exhibits a superionic-like helium diffusive state at the high pressure and high temperature conditions along the isentropes of Saturn, a metallic fluid state in Jupiter, and a solid state in the deep interiors of Uranus and Neptune. The reaction of helium and silica may lead to the erosion of the rocky core of giant planets and form a diluted core region. These results highlight the reactivity of helium under high pressure to form new compounds, and also provides evidence to help build more sophisticated interior models of giant planets.

cond-mat.mtrl-sci↗

arXiv2021

cij: A Python code for quasiharmonic thermoelasticity

The Wu-Wentzcovitch semi-analytical method (SAM) is a concise and predictive formalism to calculate the high-pressure and high-temperature (high-PT) thermoelastic tensor (Cij) of crystalline materials. This method has been successfully applied to materials across different crystal systems in conjunction with ab initio calculations of static elastic coefficients and phonon frequencies. Such results have offered first-hand insights into the composition and structure of the Earth's mantle. Here we introduce the cij package, a Python implementation of the SAM-Cij formalism. It enables a thermoelasticity calculation to be initiated from a single command and fully configurable from a calculation settings file to work with solids within any crystalline system. These features allow SAM-Cij calculations to work on a personal computer and to be easily integrated as a part of high-throughput workflows. Here we show the performance of this code for three minerals from different crystal systems at their relevant PTs: diopside (monoclinic), akimotoite (trigonal), and bridgmanite (orthorhombic).

physics.comp-ph↗

arXiv2020

Heavy dark matter and Gravitational waves

Domain walls can form after breakdown of a discrete symmetry induced by first-order phase transition, we study the heavy dark matter produced around the temperature of the phase transition that yields the breakdown of a $\mathbb{Z}_{3}$ symmetry. The generated gravitational waves by domain walls decay is found to be able to probed by the Pulsar Timing Arrays, and the future Square Kilometer Array.

hep-ph↗

arXiv2020

Multi-level Wavelet-based Generative Adversarial Network for Perceptual Quality Enhancement of Compressed Video

The past few years have witnessed fast development in video quality enhancement via deep learning. Existing methods mainly focus on enhancing the objective quality of compressed video while ignoring its perceptual quality. In this paper, we focus on enhancing the perceptual quality of compressed video. Our main observation is that enhancing the perceptual quality mostly relies on recovering high-frequency sub-bands in wavelet domain. Accordingly, we propose a novel generative adversarial network (GAN) based on multi-level wavelet packet transform (WPT) to enhance the perceptual quality of compressed video, which is called multi-level wavelet-based GAN (MW-GAN). In MW-GAN, we first apply motion compensation with a pyramid architecture to obtain temporal information. Then, we propose a wavelet reconstruction network with wavelet-dense residual blocks (WDRB) to recover the high-frequency details. In addition, the adversarial loss of MW-GAN is added via WPT to further encourage high-frequency details recovery for video frames. Experimental results demonstrate the superiority of our method.

eess.IV↗

arXiv2020

Semi-Supervised Learning Approach to Discover Enterprise User Insights from Feedback and Support

With the evolution of the cloud and customer centric culture, we inherently accumulate huge repositories of textual reviews, feedback, and support data.This has driven enterprises to seek and research engagement patterns, user network analysis, topic detections, etc.However, huge manual work is still necessary to mine data to be able to mine actionable outcomes. In this paper, we proposed and developed an innovative Semi-Supervised Learning approach by utilizing Deep Learning and Topic Modeling to have a better understanding of the user voice.This approach combines a BERT-based multiclassification algorithm through supervised learning combined with a novel Probabilistic and Semantic Hybrid Topic Inference (PSHTI) Model through unsupervised learning, aiming at automating the process of better identifying the main topics or areas as well as the sub-topics from the textual feedback and support.There are three major break-through: 1. As the advancement of deep learning technology, there have been tremendous innovations in the NLP field, yet the traditional topic modeling as one of the NLP applications lag behind the tide of deep learning. In the methodology and technical perspective, we adopt transfer learning to fine-tune a BERT-based multiclassification system to categorize the main topics and then utilize the novel PSHTI model to infer the sub-topics under the predicted main topics. 2. The traditional unsupervised learning-based topic models or clustering methods suffer from the difficulty of automatically generating a meaningful topic label, but our system enables mapping the top words to the self-help issues by utilizing domain knowledge about the product through web-crawling. 3. This work provides a prominent showcase by leveraging the state-of-the-art methodology in the real production to help shed light to discover user insights and drive business investment priorities.

cs.LG↗

arXiv2019

Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion

In this paper, we propose a novel deep convolutional neural network to solve the general multi-modal image restoration (MIR) and multi-modal image fusion (MIF) problems. Different from other methods based on deep learning, our network architecture is designed by drawing inspirations from a new proposed multi-modal convolutional sparse coding (MCSC) model. The key feature of the proposed network is that it can automatically split the common information shared among different modalities, from the unique information that belongs to each single modality, and is therefore denoted with CU-Net, i.e., Common and Unique information splitting network. Specifically, the CU-Net is composed of three modules, i.e., the unique feature extraction module (UFEM), common feature preservation module (CFPM), and image reconstruction module (IRM). The architecture of each module is derived from the corresponding part in the MCSC model, which consists of several learned convolutional sparse coding (LCSC) blocks. Extensive numerical results verify the effectiveness of our method on a variety of MIR and MIF tasks, including RGB guided depth image super-resolution, flash guided non-flash image denoising, multi-focus and multi-exposure image fusion.

cs.CV↗

arXiv2019

Wavelet Domain Style Transfer for an Effective Perception-distortion Tradeoff in Single Image Super-Resolution

In single image super-resolution (SISR), given a low-resolution (LR) image, one wishes to find a high-resolution (HR) version of it which is both accurate and photo-realistic. Recently, it has been shown that there exists a fundamental tradeoff between low distortion and high perceptual quality, and the generative adversarial network (GAN) is demonstrated to approach the perception-distortion (PD) bound effectively. In this paper, we propose a novel method based on wavelet domain style transfer (WDST), which achieves a better PD tradeoff than the GAN based methods. Specifically, we propose to use 2D stationary wavelet transform (SWT) to decompose one image into low-frequency and high-frequency sub-bands. For the low-frequency sub-band, we improve its objective quality through an enhancement network. For the high-frequency sub-band, we propose to use WDST to effectively improve its perceptual quality. By feat of the perfect reconstruction property of wavelets, these sub-bands can be re-combined to obtain an image which has simultaneously high objective and perceptual quality. The numerical results on various datasets show that our method achieves the best trade-off between the distortion and perceptual quality among the existing state-of-the-art SISR methods.

eess.IV↗

arXiv2018

Electroweak phase transition and Higgs phenomenology in the Georgi-Machacek model

In this work, we perform the electroweak phase transition study with the Georgi-Machacek model. We investigate both the one-step and two-step strong first order electroweak phase transition (SFOEWPT). The SFOEWPT viable parameter spaces could be tested by the future 14 TeV LHC, HL-LHC, and ILC. The LHC Higgs signal strength measurements severely bound the SFOEWPT valid parameter spaces, a tinny region of the mixing angle between the neutral fields of the isospin-doublet and isospin-triplet scalars around $α\sim 0$ can allow the two-step SFOEWPT to occur. The triplet vacuum expectation value (VEV) is crucial for both SFOEWPT and related Higgs phenomenology. The two-step SFOEWPT can be distinguished from the one-step SFOEWPT through the triple Higgs coupling searches and the low mass doubly charged Higgs searches at colliders.

hep-ph↗

arXiv2018

Reducing Complexity of HEVC: A Deep Learning Approach

High Efficiency Video Coding (HEVC) significantly reduces bit-rates over the proceeding H.264 standard but at the expense of extremely high encoding complexity. In HEVC, the quad-tree partition of coding unit (CU) consumes a large proportion of the HEVC encoding complexity, due to the bruteforce search for rate-distortion optimization (RDO). Therefore, this paper proposes a deep learning approach to predict the CU partition for reducing the HEVC complexity at both intra- and inter-modes, which is based on convolutional neural network (CNN) and long- and short-term memory (LSTM) network. First, we establish a large-scale database including substantial CU partition data for HEVC intra- and inter-modes. This enables deep learning on the CU partition. Second, we represent the CU partition of an entire coding tree unit (CTU) in the form of a hierarchical CU partition map (HCPM). Then, we propose an early-terminated hierarchical CNN (ETH-CNN) for learning to predict the HCPM. Consequently, the encoding complexity of intra-mode HEVC can be drastically reduced by replacing the brute-force search with ETH-CNN to decide the CU partition. Third, an early-terminated hierarchical LSTM (ETH-LSTM) is proposed to learn the temporal correlation of the CU partition. Then, we combine ETH-LSTM and ETH-CNN to predict the CU partition for reducing the HEVC complexity for inter-mode. Finally, experimental results show that our approach outperforms other state-of-the-art approaches in reducing the HEVC complexity at both intra- and inter-modes.

cs.CV↗