Searcharxiv⌕ Search

arXiv subjects

Haotian Wang

Publications and source records attributed to Haotian Wang.

At least 109 records · Page 6Linked to original sources
arXiv2024

A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames, performing even worse than single-modality models. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. In this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason. Moreover, we present the Modality Bias Hypothesis (MBH) to systematically describe the relationship between modality bias and robustness against missing modality in multimodal systems. Building on these findings, we propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality and to maintain performance and robustness simultaneously. Finally, to address an entirely missing modality, we adopt adapters to dynamically switch decision strategies. The effectiveness of our proposed approach is evaluated and validated through a series of comprehensive experiments using the MISP2021 and MISP2022 datasets. Our code is available at https://github.com/dalision/ModalBiasAVSR

cs.SD↗
arXiv2024

Flares hunting in hot subdwarf and white dwarf stars from Cycles 1-5 of TESS photometry

Stellar flares are critical phenomena on stellar surfaces, which are closely tied to stellar magnetism. While extensively studied in main-sequence (MS) stars, their occurrence in evolved compact stars, specifically hot subdwarfs and white dwarfs (WDs), remains scarcely explored. Based on Cycles 1-5 of TESS photometry, we conducted a pioneering survey of flare events in $\sim12,000$ compact stars, corresponding to $\sim38,000$ light curves with 2-minute cadence. Through dedicated techniques for detrending light curves, identifying preliminary flare candidates, and validating them via machine learning, we established a catalog of 1016 flares from 193 compact stars, including 182 from 58 sdB/sdO stars and 834 from 135 WDs, respectively. However, all flaring compact stars showed signs of contamination from nearby objects or companion stars, preventing sole attribution of the detected flares. For WDs, it is highly probable that the flares originated from their cool MS companions. In contrast, the higher luminosities of sdB/sdO stars diminish companion contributions, suggesting that detected flares originated from sdB/sdO stars themselves or through close magnetic interactions with companions. Focusing on a refined sample of 23 flares from 13 sdB/sdO stars, we found their flare frequency distributions were slightly divergent from those of cool MS stars; instead, they resemble those of hot B/A-type MS stars having radiative envelopes. This similarity implies the flares on sdB/sdO stars, if these flares did originate from them, may share underlying mechanisms with hot MS stars, which warrants further investigation.

astro-ph.SR↗
arXiv2024

Composite Active Learning: Towards Multi-Domain Active Learning with Theoretical Guarantees

Active learning (AL) aims to improve model performance within a fixed labeling budget by choosing the most informative data points to label. Existing AL focuses on the single-domain setting, where all data come from the same domain (e.g., the same dataset). However, many real-world tasks often involve multiple domains. For example, in visual recognition, it is often desirable to train an image classifier that works across different environments (e.g., different backgrounds), where images from each environment constitute one domain. Such a multi-domain AL setting is challenging for prior methods because they (1) ignore the similarity among different domains when assigning labeling budget and (2) fail to handle distribution shift of data across different domains. In this paper, we propose the first general method, dubbed composite active learning (CAL), for multi-domain AL. Our approach explicitly considers the domain-level and instance-level information in the problem; CAL first assigns domain-level budgets according to domain-level importance, which is estimated by optimizing an upper error bound that we develop; with the domain-level budgets, CAL then leverages a certain instance-level query strategy to select samples to label from each domain. Our theoretical analysis shows that our method achieves a better error bound compared to current AL methods. Our empirical results demonstrate that our approach significantly outperforms the state-of-the-art AL methods on both synthetic and real-world multi-domain datasets. Code is available at https://github.com/Wang-ML-Lab/multi-domain-active-learning.

cs.LG↗
arXiv2023

Laser frequency stabilization and photoacoustic detection based on the tapered fiber coupled crystalline resonator

We demonstrate laser frequency stabilization using a high-Q MgF2 crystalline whispering gallery mode resonator coupled with a tapered fiber. We discovered that the tapered fiber, acting as a microcantilever, exhibits mechanical resonance characteristics that is capable of transmitting acoustic perturbations to the frequency locking loop. Both experimental and theoretical investigations into the influence of external acoustic waves on the coupling system were conducted. After acoustic isolation, the locked laser exhibits a minimum frequency noise of 0.4Hz2/Hz at 7kHz and an integral linewidth of 68Hz (0.1s integration time). Benefiting from the ultralow frequency noise of the stabilized laser, it achieves a minimum noise equivalent acoustic signal level of 4.76*10-4 Pa/Hz1/2. Our results not only facilitate the realization of ultralow noise lasers but also serves as a novel and sensitive photoacoustic detector.

physics.optics↗
arXiv2023

G2-MonoDepth: A General Framework of Generalized Depth Inference from Monocular RGB+X Data

Monocular depth inference is a fundamental problem for scene perception of robots. Specific robots may be equipped with a camera plus an optional depth sensor of any type and located in various scenes of different scales, whereas recent advances derived multiple individual sub-tasks. It leads to additional burdens to fine-tune models for specific robots and thereby high-cost customization in large-scale industrialization. This paper investigates a unified task of monocular depth inference, which infers high-quality depth maps from all kinds of input raw data from various robots in unseen scenes. A basic benchmark G2-MonoDepth is developed for this task, which comprises four components: (a) a unified data representation RGB+X to accommodate RGB plus raw depth with diverse scene scale/semantics, depth sparsity ([0%, 100%]) and errors (holes/noises/blurs), (b) a novel unified loss to adapt to diverse depth sparsity/errors of input raw data and diverse scales of output scenes, (c) an improved network to well propagate diverse scene scales from input to output, and (d) a data augmentation pipeline to simulate all types of real artifacts in raw depth maps for training. G2-MonoDepth is applied in three sub-tasks including depth estimation, depth completion with different sparsity, and depth enhancement in unseen scenes, and it always outperforms SOTA baselines on both real-world data and synthetic data.

cs.CV↗
arXiv2023

Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023

In this paper, we propose a novel framework for recognizing both discrete and dimensional emotions. In our framework, deep features extracted from foundation models are used as robust acoustic and visual representations of raw video. Three different structures based on attention-guided feature gathering (AFG) are designed for deep feature fusion. Then, we introduce a joint decoding structure for emotion classification and valence regression in the decoding stage. A multi-task loss based on uncertainty is also designed to optimize the whole process. Finally, by combining three different structures on the posterior probability level, we obtain the final predictions of discrete and dimensional emotions. When tested on the dataset of multimodal emotion recognition challenge (MER 2023), the proposed framework yields consistent improvements in both emotion classification and valence regression. Our final system achieves state-of-the-art performance and ranks third on the leaderboard on MER-MULTI sub-challenge.

eess.AS↗
arXiv2023

Competing for Shareable Arms in Multi-Player Multi-Armed Bandits

Competitions for shareable and limited resources have long been studied with strategic agents. In reality, agents often have to learn and maximize the rewards of the resources at the same time. To design an individualized competing policy, we model the competition between agents in a novel multi-player multi-armed bandit (MPMAB) setting where players are selfish and aim to maximize their own rewards. In addition, when several players pull the same arm, we assume that these players averagely share the arms' rewards by expectation. Under this setting, we first analyze the Nash equilibrium when arms' rewards are known. Subsequently, we propose a novel Selfish MPMAB with Averaging Allocation (SMAA) approach based on the equilibrium. We theoretically demonstrate that SMAA could achieve a good regret guarantee for each player when all players follow the algorithm. Additionally, we establish that no single selfish player can significantly increase their rewards through deviation, nor can they detrimentally affect other players' rewards without incurring substantial losses for themselves. We finally validate the effectiveness of the method in extensive synthetic experiments.

cs.LG↗
arXiv2023

Breast Ultrasound Tumor Classification Using a Hybrid Multitask CNN-Transformer Network

Capturing global contextual information plays a critical role in breast ultrasound (BUS) image classification. Although convolutional neural networks (CNNs) have demonstrated reliable performance in tumor classification, they have inherent limitations for modeling global and long-range dependencies due to the localized nature of convolution operations. Vision Transformers have an improved capability of capturing global contextual information but may distort the local image patterns due to the tokenization operations. In this study, we proposed a hybrid multitask deep neural network called Hybrid-MT-ESTAN, designed to perform BUS tumor classification and segmentation using a hybrid architecture composed of CNNs and Swin Transformer components. The proposed approach was compared to nine BUS classification methods and evaluated using seven quantitative metrics on a dataset of 3,320 BUS images. The results indicate that Hybrid-MT-ESTAN achieved the highest accuracy, sensitivity, and F1 score of 82.7%, 86.4%, and 86.0%, respectively.

eess.IV↗
arXiv2023

Transactional Indexes on (RDMA or CXL-based) Disaggregated Memory with Repairable Transaction

The failure atomic and isolated execution of clients operations is a default requirement for a system that serve multiple loosely coupled clients at a server. However, disaggregated memory breaks this requirement in remote indexes because a client operation is disaggregated to multiple remote reads/writes. Current indexes focus on performance improvements and largely ignore tolerating client failures. We argue that a practical DM index should be transactional: each index operation should be failure atomic and isolated in addition to being concurrency isolated. We present repairable transaction (rTX), a lightweight primitive to execute DM index operations. Each rTX can detect other failed rTXes on-the-fly with the help of concurrency control. Upon detection, it will repair their non-atomic updates online with the help of logging, thus hiding their failures from healthy clients. By further removing unnecessary logging and delegating concurrency control to existing carefully-tuned index algorithms, we show that transactional indexes can be built at a low performance overhead on disaggregated memory. We have refactored two state-of-the-art DM indexes, RaceHashing and Sherman (B+Tree), with rTX. Evaluations show that rTX is 1.2 to 2X faster than other alternatives, e.g., distributed transaction. Meanwhile, its overhead is up to 42% compared to non-fault-tolerant indexes.

cs.DB↗
arXiv2023

Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning

Code comment generation aims at generating natural language descriptions for a code snippet to facilitate developers' program comprehension activities. Despite being studied for a long time, a bottleneck for existing approaches is that given a code snippet, they can only generate one comment while developers usually need to know information from diverse perspectives such as what is the functionality of this code snippet and how to use it. To tackle this limitation, this study empirically investigates the feasibility of utilizing large language models (LLMs) to generate comments that can fulfill developers' diverse intents. Our intuition is based on the facts that (1) the code and its pairwise comment are used during the pre-training process of LLMs to build the semantic connection between the natural language and programming language, and (2) comments in the real-world projects, which are collected for the pre-training, usually contain different developers' intents. We thus postulate that the LLMs can already understand the code from different perspectives after the pre-training. Indeed, experiments on two large-scale datasets demonstrate the rationale of our insights: by adopting the in-context learning paradigm and giving adequate prompts to the LLM (e.g., providing it with ten or more examples), the LLM can significantly outperform a state-of-the-art supervised learning approach on generating comments with multiple intents. Results also show that customized strategies for constructing the prompts and post-processing strategies for reranking the results can both boost the LLM's performances, which shed light on future research directions for using LLMs to achieve comment generation.

cs.SE↗
arXiv2023

E-NER: Evidential Deep Learning for Trustworthy Named Entity Recognition

Most named entity recognition (NER) systems focus on improving model performance, ignoring the need to quantify model uncertainty, which is critical to the reliability of NER systems in open environments. Evidential deep learning (EDL) has recently been proposed as a promising solution to explicitly model predictive uncertainty for classification tasks. However, directly applying EDL to NER applications faces two challenges, i.e., the problems of sparse entities and OOV/OOD entities in NER tasks. To address these challenges, we propose a trustworthy NER framework named E-NER by introducing two uncertainty-guided loss terms to the conventional EDL, along with a series of uncertainty-guided training strategies. Experiments show that E-NER can be applied to multiple NER paradigms to obtain accurate uncertainty estimation. Furthermore, compared to state-of-the-art baselines, the proposed method achieves a better OOV/OOD detection performance and better generalization ability on OOV entities.

cs.CL↗
arXiv2023

SIAN: Style-Guided Instance-Adaptive Normalization for Multi-Organ Histopathology Image Synthesis

Existing deep neural networks for histopathology image synthesis cannot generate image styles that align with different organs, and cannot produce accurate boundaries of clustered nuclei. To address these issues, we propose a style-guided instance-adaptive normalization (SIAN) approach to synthesize realistic color distributions and textures for histopathology images from different organs. SIAN contains four phases, semantization, stylization, instantiation, and modulation. The first two phases synthesize image semantics and styles by using semantic maps and learned image style vectors. The instantiation module integrates geometrical and topological information and generates accurate nuclei boundaries. We validate the proposed approach on a multiple-organ dataset, Extensive experimental results demonstrate that the proposed method generates more realistic histopathology images than four state-of-the-art approaches for five organs. By incorporating synthetic images from the proposed approach to model training, an instance segmentation network can achieve state-of-the-art performance.

eess.IV↗
arXiv2023

Enhanced Sharp-GAN For Histopathology Image Synthesis

Histopathology image synthesis aims to address the data shortage issue in training deep learning approaches for accurate cancer detection. However, existing methods struggle to produce realistic images that have accurate nuclei boundaries and less artifacts, which limits the application in downstream tasks. To address the challenges, we propose a novel approach that enhances the quality of synthetic images by using nuclei topology and contour regularization. The proposed approach uses the skeleton map of nuclei to integrate nuclei topology and separate touching nuclei. In the loss function, we propose two new contour regularization terms that enhance the contrast between contour and non-contour pixels and increase the similarity between contour pixels. We evaluate the proposed approach on the two datasets using image quality metrics and a downstream task (nuclei segmentation). The proposed approach outperforms Sharp-GAN in all four image quality metrics on two datasets. By integrating 6k synthetic images from the proposed approach into training, a nuclei segmentation model achieves the state-of-the-art segmentation performance on TNBC dataset and its detection quality (DQ), segmentation quality (SQ), panoptic quality (PQ), and aggregated Jaccard index (AJI) is 0.855, 0.863, 0.691, and 0.683, respectively.

eess.IV↗
arXiv2022

Stacking order and interlayer coupling tuning the properties of charge density waves in layered 1T-NbSe_2

Layered transition metal dichalcogenide 1T-NbSe_2 is a good candidate to explore the charge density wave (CDW) and Mott physics. However, the effects of stacking orders and interlayer coupling in CDW 1T-NbSe_2 are still less explored and understood. Using density functional theory calculations, we present a systematic study of the electronic and magnetic properties of monolayer and layered CDW 1T-NbSe_2. Our results indicate that monolayer CDW 1T-NbSe_2 is a magnetic insulator with \sqrt13\times\sqrt13 periodic lattice modulation. Nevertheless, the magnetic properties of bilayer CDWs 1T-NbSe_2 are found stacking orders dependence. The mechanism is understood by the changes of local magnetic moments in each layer due to spin charge transfer between interlayers. Furthermore, the bulk CDW 1T-NbSe_2 opens a band gap with 0.02 eV in 1\times 1 \times 2 supercell due to the interlayer spin coupling. We also discover that the electronic structures of layered 1T-NbSe_2 show a strong dependence on stacking configurations and dimensionality.

cond-mat.str-el↗
arXiv2022

On Cyclic Solutions to the Min-Max Latency Multi-Robot Patrolling Problem

We consider the following surveillance problem: Given a set $P$ of $n$ sites in a metric space and a set of $k$ robots with the same maximum speed, compute a patrol schedule of minimum latency for the robots. Here a patrol schedule specifies for each robot an infinite sequence of sites to visit (in the given order) and the latency $L$ of a schedule is the maximum latency of any site, where the latency of a site $s$ is the supremum of the lengths of the time intervals between consecutive visits to $s$. When $k=1$ the problem is equivalent to the travelling salesman problem (TSP) and thus it is NP-hard. We have two main results. We consider cyclic solutions in which the set of sites must be partitioned into $\ell$ groups, for some~$\ell \leq k$, and each group is assigned a subset of the robots that move along the travelling salesman tour of the group at equal distance from each other. Our first main result is that approximating the optimal latency of the class of cyclic solutions can be reduced to approximating the optimal travelling salesman tour on some input, with only a $1+\varepsilon$ factor loss in the approximation factor and an $O\left(\left( k/\varepsilon \right)^k\right)$ factor loss in the runtime, for any $\varepsilon >0$. Our second main result shows that an optimal cyclic solution is a $2(1-1/k)$-approximation of the overall optimal solution. Note that for $k=2$ this implies that an optimal cyclic solution is optimal overall. The results have a number of consequences. For the Euclidean version of the problem, for instance, combining our results with known results on Euclidean TSP, yields a PTAS for approximating an optimal cyclic solution, and it yields a $(2(1-1/k)+\varepsilon)$-approximation of the optimal unrestricted solution. If the conjecture mentioned above is true, then our algorithm is actually a PTAS for the general problem in the Euclidean setting.

cs.CG↗
arXiv2021

Co-evolution of Opinion and Social Tie Dynamics Towards Structural Balance

In this paper, we propose co-evolution models for both dynamics of opinions (people's view on a particular topic) and dynamics of social appraisals (the approval or disapproval towards each other). Opinion dynamics and dynamics of signed networks, respectively, have been extensively studied. We propose a co-evolution model, where each vertex $i$ in the network has a current opinion vector $v_i$ and each edge $(i,j)$ has a weight $w_{ij}$ that models the relationship between $i,j$. The system evolves as the opinions and edge weights are updated over time by the following rules, Opinion Dynamics and Appraisal Dynamics. We are interested in characterizing the long-time behavior of the dynamic model -- i.e., whether edge weights evolve to have stable signs (positive or negative) and structural balance (the multiplication of weights on any triangle is non-negative). Our main theoretical result solves the above dynamic system with time-evolving opinions $V(t)=[v1(t),\cdots,vn(t)]$ and social tie weights $W(t)=[w_{ij}(t)]_{n\times n}$. For a generic initial opinion vector $V(0)$ and weight matrix $W(0)$, one of the two phenomena must occur at the limit. The first one is that both sign stability and structural balance (for any triangle with individual $i,j,k$, $w_{ij}w_{jk}w_{ki}\geq0$) occur. In the special case that $V(0)$ is an eigenvector of $W(0)$, we are able to obtain the explicit solution to the co-evolution equation and give exact estimates on the blowup time and rate convergence. The second one is that all the opinions converge to $0$, i.e., $\lim_{t\rightarrow \infty}|V(t)|=0$.

cs.SI↗
arXiv2021

Potato Crop Stress Identification in Aerial Images using Deep Learning-based Object Detection

Recent research on the application of remote sensing and deep learning-based analysis in precision agriculture demonstrated a potential for improved crop management and reduced environmental impacts of agricultural production. Despite the promising results, the practical relevance of these technologies for field deployment requires novel algorithms that are customized for analysis of agricultural images and robust to implementation on natural field imagery. The paper presents an approach for analyzing aerial images of a potato (Solanum tuberosum L.) crop using deep neural networks. The main objective is to demonstrate automated spatial recognition of healthy vs. stressed crop at a plant level. Specifically, we examine premature plant senescence resulting in drought stress on Russet Burbank potato plants. We propose a novel deep learning (DL) model for detecting crop stress, named Retina-UNet-Ag. The proposed architecture is a variant of Retina-UNet and includes connections from low-level semantic representation maps to the feature pyramid network. The paper also introduces a dataset of aerial field images acquired with a Parrot Sequoia camera. The dataset includes manually annotated bounding boxes of healthy and stressed plant regions. Experimental validation demonstrated the ability for distinguishing healthy and stressed plants in field images, achieving an average dice score coefficient (DSC) of 0.74. A comparison to related state-of-the-art DL models for object detection revealed that the presented approach is effective for this task. The proposed method is conducive toward the assessment and recognition of potato crop stress in aerial field images collected under natural conditions.

cs.CV↗
arXiv2021

TA-Net: Topology-Aware Network for Gland Segmentation

Gland segmentation is a critical step to quantitatively assess the morphology of glands in histopathology image analysis. However, it is challenging to separate densely clustered glands accurately. Existing deep learning-based approaches attempted to use contour-based techniques to alleviate this issue but only achieved limited success. To address this challenge, we propose a novel topology-aware network (TA-Net) to accurately separate densely clustered and severely deformed glands. The proposed TA-Net has a multitask learning architecture and enhances the generalization of gland segmentation by learning shared representation from two tasks: instance segmentation and gland topology estimation. The proposed topology loss computes gland topology using gland skeletons and markers. It drives the network to generate segmentation results that comply with the true gland topology. We validate the proposed approach on the GlaS and CRAG datasets using three quantitative metrics, F1-score, object-level Dice coefficient, and object-level Hausdorff distance. Extensive experiments demonstrate that TA-Net achieves state-of-the-art performance on the two datasets. TA-Net outperforms other approaches in the presence of densely clustered glands.

eess.IV↗