arXiv subjects

Jialu Li

Publications and source records attributed to Jialu Li.

At least 55 records · Page 3Linked to original sources

arXiv2024

SparrowVQE: Visual Question Explanation for Course Content Understanding

Visual Question Answering (VQA) research seeks to create AI systems to answer natural language questions in images, yet VQA methods often yield overly simplistic and short answers. This paper aims to advance the field by introducing Visual Question Explanation (VQE), which enhances the ability of VQA to provide detailed explanations rather than brief responses and address the need for more complex interaction with visual content. We first created an MLVQE dataset from a 14-week streamed video machine learning course, including 885 slide images, 110,407 words of transcripts, and 9,416 designed question-answer (QA) pairs. Next, we proposed a novel SparrowVQE, a small 3 billion parameters multimodal model. We trained our model with a three-stage training mechanism consisting of multimodal pre-training (slide images and transcripts feature alignment), instruction tuning (tuning the pre-trained model with transcripts and QA pairs), and domain fine-tuning (fine-tuning slide image and QA pairs). Eventually, our SparrowVQE can understand and connect visual information using the SigLIP model with transcripts using the Phi-2 language model with an MLP adapter. Experimental results demonstrate that our SparrowVQE achieves better performance in our developed MLVQE dataset and outperforms state-of-the-art methods in the other five benchmark VQA datasets. The source code is available at \url{https://github.com/YoushanZhang/SparrowVQE}.

cs.CV↗

arXiv2024

Unbounded: A Generative Infinite Game of Character Life Simulation

We introduce the concept of a generative infinite game, a video game that transcends the traditional boundaries of finite, hard-coded systems by using generative models. Inspired by James P. Carse's distinction between finite and infinite games, we leverage recent advances in generative AI to create Unbounded: a game of character life simulation that is fully encapsulated in generative models. Specifically, Unbounded draws inspiration from sandbox life simulations and allows you to interact with your autonomous virtual character in a virtual world by feeding, playing with and guiding it - with open-ended mechanics generated by an LLM, some of which can be emergent. In order to develop Unbounded, we propose technical innovations in both the LLM and visual generation domains. Specifically, we present: (1) a specialized, distilled large language model (LLM) that dynamically generates game mechanics, narratives, and character interactions in real-time, and (2) a new dynamic regional image prompt Adapter (IP-Adapter) for vision models that ensures consistent yet flexible visual generation of a character across multiple environments. We evaluate our system through both qualitative and quantitative analysis, showing significant improvements in character life simulation, user instruction following, narrative coherence, and visual consistency for both characters and the environments compared to traditional related approaches.

cs.CV↗

arXiv2024

A quasi-ohmic back contact achieved by inserting single-crystal graphene in flexible Kesterite solar cells

Flexible photovoltaics with a lightweight and adaptable nature that allows for deployment on curved surfaces and in building facades have always been a goal vigorously pursued by researchers in thin-film solar cell technology. The recent strides made in improving the sunlight-to-electricity conversion efficiency of kesterite Cu$_{2}$ZnSn(S, Se)$_{4}$ (CZTSSe) suggest it to be a perfect candidate. However, making use of rare Mo foil in CZTSSe solar cells causes severe problems in thermal expansion matching, uneven grain growth, and severe problems at the back contact of the devices. Herein, a strategy utilizing single-crystal graphene to modify the back interface of flexible CZTSSe solar cells is proposed. It will be shown that the insertion of graphene at the Mo foil/CZTSSe interface provides strong physical support for the subsequent deposition of the CZTSSe absorber layer, improving the adhesion between the absorber layer and the Mo foil substrate. Additionally, the graphene passivates the rough sites on the surface of the Mo foil, enhancing the chemical homogeneity of the substrate, and resulting in a more crystalline and homogeneous CZTSSe absorber layer on the Mo foil substrate. The detrimental reaction between Mo and CZTSSe has also been eliminated. Through an analysis of the electrical properties, it is found that the introduction of graphene at the back interface promotes the formation of a quasi-ohmic contact at the back contact, decreasing the back contact barrier of the solar cell, and leading to efficient collection of charges at the back interface. This investigation demonstrates that solution-based CZTSSe photovoltaic devices could form the basis of cheap and flexible solar cells.

cond-mat.mtrl-sci↗

arXiv2024

Sound Tagging in Infant-centric Home Soundscapes

Certain environmental noises have been associated with negative developmental outcomes for infants and young children. Though classifying or tagging sound events in a domestic environment is an active research area, previous studies focused on data collected from a non-stationary microphone placed in the environment or from the perspective of adults. Further, many of these works ignore infants or young children in the environment or have data collected from only a single family where noise from the fixed sound source can be moderate at the infant's position or vice versa. Thus, despite the recent success of large pre-trained models for noise event detection, the performance of these models on infant-centric noise soundscapes in the home is yet to be explored. To bridge this gap, we have collected and labeled noises in home soundscapes from 22 families in an unobtrusive manner, where the data are collected through an infant-worn recording device. In this paper, we explore the performance of a large pre-trained model (Audio Spectrogram Transformer [AST]) on our noise-conditioned infant-centric environmental data as well as publicly available home environmental datasets. Utilizing different training strategies such as resampling, utilizing public datasets, mixing public and infant-centric training sets, and data augmentation using noise and masking, we evaluate the performance of a large pre-trained model on sparse and imbalanced infant-centric data. Our results show that fine-tuning the large pre-trained model by combining our collected dataset with public datasets increases the F1-score from 0.11 (public datasets) and 0.76 (collected datasets) to 0.84 (combined datasets) and Cohen's Kappa from 0.013 (public datasets) and 0.77 (collected datasets) to 0.83 (combined datasets) compared to only training with public or collected datasets, respectively.

cs.SD↗

arXiv2024

QUADFormer: Learning-based Detection of Cyber Attacks in Quadrotor UAVs

Safety-critical intelligent cyber-physical systems, such as quadrotor unmanned aerial vehicles (UAVs), are vulnerable to different types of cyber attacks, and the absence of timely and accurate attack detection can lead to severe consequences. When UAVs are engaged in large outdoor maneuvering flights, their system constitutes highly nonlinear dynamics that include non-Gaussian noises. Therefore, the commonly employed traditional statistics-based and emerging learning-based attack detection methods do not yield satisfactory results. In response to the above challenges, we propose QUADFormer, a novel Quadrotor UAV Attack Detection framework with transFormer-based architecture. This framework includes a residue generator designed to generate a residue sequence sensitive to anomalies. Subsequently, this sequence is fed into a transformer structure with disparity in correlation to specifically learn its statistical characteristics for the purpose of classification and attack detection. Finally, we design an alert module to ensure the safe execution of tasks by UAVs under attack conditions. We conduct extensive simulations and real-world experiments, and the results show that our method has achieved superior detection performance compared with many state-of-the-art methods.

cs.RO↗

arXiv2024

Diffusion Gaussian Mixture Audio Denoise

Recent diffusion models have achieved promising performances in audio-denoising tasks. The unique property of the reverse process could recover clean signals. However, the distribution of real-world noises does not comply with a single Gaussian distribution and is even unknown. The sampling of Gaussian noise conditions limits its application scenarios. To overcome these challenges, we propose a DiffGMM model, a denoising model based on the diffusion and Gaussian mixture models. We employ the reverse process to estimate parameters for the Gaussian mixture model. Given a noisy audio signal, we first apply a 1D-U-Net to extract features and train linear layers to estimate parameters for the Gaussian mixture model, and we approximate the real noise distributions. The noisy signal is continuously subtracted from the estimated noise to output clean audio signals. Extensive experimental results demonstrate that the proposed DiffGMM model achieves state-of-the-art performance.

cs.SD↗

arXiv2024

Complex Image-Generative Diffusion Transformer for Audio Denoising

The audio denoising technique has captured widespread attention in the deep neural network field. Recently, the audio denoising problem has been converted into an image generation task, and deep learning-based approaches have been applied to tackle this problem. However, its performance is still limited, leaving room for further improvement. In order to enhance audio denoising performance, this paper introduces a complex image-generative diffusion transformer that captures more information from the complex Fourier domain. We explore a novel diffusion transformer by integrating the transformer with a diffusion model. Our proposed model demonstrates the scalability of the transformer and expands the receptive field of sparse attention using attention diffusion. Our work is among the first to utilize diffusion transformers to deal with the image generation task for audio denoising. Extensive experiments on two benchmark datasets demonstrate that our proposed model outperforms state-of-the-art methods.

cs.SD↗

arXiv2024

Vision Transformer Segmentation for Visual Bird Sound Denoising

Audio denoising, especially in the context of bird sounds, remains a challenging task due to persistent residual noise. Traditional and deep learning methods often struggle with artificial or low-frequency noise. In this work, we propose ViTVS, a novel approach that leverages the power of the vision transformer (ViT) architecture. ViTVS adeptly combines segmentation techniques to disentangle clean audio from complex signal mixtures. Our key contributions encompass the development of ViTVS, introducing comprehensive, long-range, and multi-scale representations. These contributions directly tackle the limitations inherent in conventional approaches. Extensive experiments demonstrate that ViTVS outperforms state-of-the-art methods, positioning it as a benchmark solution for real-world bird sound denoising applications. Source code is available at: https://github.com/aiai-4/ViVTS.

cs.SD↗

arXiv2024

On the Interpretation of Mid-Infrared Absorption Lines of Gas-Phase H$_2$O as Observed by JWST/MIRI

Ro-vibrational absorption lines of H$_2$O in the 5-8 $μ$m wavelength range selectively probe gas against the mid-infrared continuum emitting background of the inner regions of YSOs and AGN and deliver important information about these warm, dust-obscured environments. JWST/MIRI detects these lines in many lines of sight at a moderate spectral resolving power of $R\sim3500$ (FWHM of 85 km/s). Based on our analysis of high-resolution SOFIA/EXES observations, we find that the interpretation of JWST/MIRI absorption spectra can be severely hampered by the blending of individual transitions and the lost information on the intrinsic line width or the partial coverage of the background continuum source. In this paper, we point out problems such as degeneracy that arise in deriving physical properties from an insufficiently resolved spectrum. This can lead to differences in the column density by two orders of magnitude. We emphasize the importance of weighting optically thin and weak lines in spectral analyses and provide recipes for breaking down the coupled parameters. We also provide an online tool to generate the H$_2$O absorption line spectra that can be compared to observations.

astro-ph.IM↗

arXiv2024

Enhancing Child Vocalization Classification with Phonetically-Tuned Embeddings for Assisting Autism Diagnosis

The assessment of children at risk of autism typically involves a clinician observing, taking notes, and rating children's behaviors. A machine learning model that can label adult and child audio may largely save labor in coding children's behaviors, helping clinicians capture critical events and better communicate with parents. In this study, we leverage Wav2Vec 2.0 (W2V2), pre-trained on 4300-hour of home audio of children under 5 years old, to build a unified system for tasks of clinician-child speaker diarization and vocalization classification (VC). To enhance children's VC, we build a W2V2 phoneme recognition system for children under 4 years old, and we incorporate its phonetically-tuned embeddings as auxiliary features or recognize pseudo phonetic transcripts as an auxiliary task. We test our method on two corpora (Rapid-ABC and BabbleCor) and obtain consistent improvements. Additionally, we outperform the state-of-the-art performance on the reproducible subset of BabbleCor. Code available at https://huggingface.co/lijialudew

eess.AS↗

arXiv2024

Analysis of Self-Supervised Speech Models on Children's Speech and Infant Vocalizations

To understand why self-supervised learning (SSL) models have empirically achieved strong performances on several speech-processing downstream tasks, numerous studies have focused on analyzing the encoded information of the SSL layer representations in adult speech. Limited work has investigated how pre-training and fine-tuning affect SSL models encoding children's speech and vocalizations. In this study, we aim to bridge this gap by probing SSL models on two relevant downstream tasks: (1) phoneme recognition (PR) on the speech of adults, older children (8-10 years old), and younger children (1-4 years old), and (2) vocalization classification (VC) distinguishing cry, fuss, and babble for infants under 14 months old. For younger children's PR, the superiority of fine-tuned SSL models is largely due to their ability to learn features that represent older children's speech and then adapt those features to the speech of younger children. For infant VC, SSL models pre-trained on large-scale home recordings learn to leverage phonetic representations at middle layers, and thereby enhance the performance of this task.

eess.AS↗

arXiv2024

SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data

Recent text-to-image (T2I) generation models have demonstrated impressive capabilities in creating images from text descriptions. However, these T2I generation models often fall short of generating images that precisely match the details of the text inputs, such as incorrect spatial relationship or missing objects. In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging. First, SELMA leverages an LLM's in-context learning capability to generate multiple datasets of text prompts that can teach different skills, and then generates the images with a T2I model based on the prompts. Next, SELMA adapts the T2I model to the new skills by learning multiple single-skill LoRA (low-rank adaptation) experts followed by expert merging. Our independent expert fine-tuning specializes multiple models for different skills, and expert merging helps build a joint multi-skill T2I model that can generate faithful images given diverse text prompts, while mitigating the knowledge conflict from different datasets. We empirically demonstrate that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-art T2I diffusion models on multiple benchmarks (+2.1% on TIFA and +6.9% on DSG), human preference metrics (PickScore, ImageReward, and HPS), as well as human evaluation. Moreover, fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data. Lastly, we show that fine-tuning with images from a weaker T2I model can help improve the generation quality of a stronger T2I model, suggesting promising weak-to-strong generalization in T2I models.

cs.CV↗

arXiv2024

VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigation

Outdoor Vision-and-Language Navigation (VLN) requires an agent to navigate through realistic 3D outdoor environments based on natural language instructions. The performance of existing VLN methods is limited by insufficient diversity in navigation environments and limited training data. To address these issues, we propose VLN-Video, which utilizes the diverse outdoor environments present in driving videos in multiple cities in the U.S. augmented with automatically generated navigation instructions and actions to improve outdoor VLN performance. VLN-Video combines the best of intuitive classical approaches and modern deep learning techniques, using template infilling to generate grounded navigation instructions, combined with an image rotation similarity-based navigation action predictor to obtain VLN style data from driving videos for pretraining deep learning VLN models. We pre-train the model on the Touchdown dataset and our video-augmented dataset created from driving videos with three proxy tasks: Masked Language Modeling, Instruction and Trajectory Matching, and Next Action Prediction, so as to learn temporally-aware and visually-aligned instruction representations. The learned instruction representation is adapted to the state-of-the-art navigator when fine-tuning on the Touchdown dataset. Empirical results demonstrate that VLN-Video significantly outperforms previous state-of-the-art models by 2.1% in task completion rate, achieving a new state-of-the-art on the Touchdown dataset.

cs.CV↗

arXiv2024

Every Node is Different: Dynamically Fusing Self-Supervised Tasks for Attributed Graph Clustering

Attributed graph clustering is an unsupervised task that partitions nodes into different groups. Self-supervised learning (SSL) shows great potential in handling this task, and some recent studies simultaneously learn multiple SSL tasks to further boost performance. Currently, different SSL tasks are assigned the same set of weights for all graph nodes. However, we observe that some graph nodes whose neighbors are in different groups require significantly different emphases on SSL tasks. In this paper, we propose to dynamically learn the weights of SSL tasks for different nodes and fuse the embeddings learned from different SSL tasks to boost performance. We design an innovative graph clustering approach, namely Dynamically Fusing Self-Supervised Learning (DyFSS). Specifically, DyFSS fuses features extracted from diverse SSL tasks using distinct weights derived from a gating network. To effectively learn the gating network, we design a dual-level self-supervised strategy that incorporates pseudo labels and the graph structure. Extensive experiments on five datasets show that DyFSS outperforms the state-of-the-art multi-task SSL methods by up to 8.66% on the accuracy metric. The code of DyFSS is available at: https://github.com/q086/DyFSS.

cs.LG↗

arXiv2023

Dynamics in Star-forming Cores (DiSCo): Project Overview and the First Look toward the B1 and NGC 1333 Regions in Perseus

The internal velocity structure within dense gaseous cores plays a crucial role in providing the initial conditions for star formation in molecular clouds. However, the kinematic properties of dense gas at core scales (~0.01 - 0.1 pc) has not been extensively characterized because of instrument limitations until the unique capabilities of GBT-Argus became available. The ongoing GBT-Argus Large Program, Dynamics in Star-forming Cores (DiSCo) thus aims to investigate the origin and distribution of angular momenta of star-forming cores. DiSCo will survey all starless cores and Class 0 protostellar cores in the Perseus molecular complex down to ~0.01 pc scales with < 0.05 km/s velocity resolution using the dense gas tracer N$_2$H$^+$. Here, we present the first datasets from DiSCo toward the B1 and NGC 1333 regions in Perseus. Our results suggest that a dense core's internal velocity structure has little correlation with other core-scale properties, indicating these gas motions may be originated externally from cloud-scale turbulence. These first datasets also reaffirm the ability of GBT-Argus for studying dense core velocity structure and provided an empirical basis for future studies that address the angular momentum problem with a statistically broad sample.

astro-ph.GA↗

arXiv2023

Towards Robust Family-Infant Audio Analysis Based on Unsupervised Pretraining of Wav2vec 2.0 on Large-Scale Unlabeled Family Audio

To perform automatic family audio analysis, past studies have collected recordings using phone, video, or audio-only recording devices like LENA, investigated supervised learning methods, and used or fine-tuned general-purpose embeddings learned from large pretrained models. In this study, we advance the audio component of a new infant wearable multi-modal device called LittleBeats (LB) by learning family audio representation via wav2vec 2.0 (W2V2) pertaining. We show given a limited number of labeled LB home recordings, W2V2 pretrained using 1k-hour of unlabeled home recordings outperforms oracle W2V2 pretrained on 960-hour unlabeled LibriSpeech in terms of parent/infant speaker diarization (SD) and vocalization classifications (VC) at home. Extra relevant external unlabeled and labeled data further benefit W2V2 pretraining and fine-tuning. With SpecAug and environmental speech corruptions, we obtain 12% relative gain on SD and moderate boost on VC. Code and model weights are available.

eess.AS↗

arXiv2023

GBT/Argus Observations of Molecular Gas in the Inner Regions of IC 342

We report observations of the ground state transitions of $^{12}$CO, $^{13}$CO C$^{18}$O, HCN, and HCO$^+$ at 88-115 GHz in the inner region of the nearby galaxy IC 342. These data were obtained with the 16-pixel spectroscopic focal plane array Argus on the 100-m Robert C. Byrd Green Bank Telescope (GBT) at 6-9$^{\prime\prime}$ resolution. In the nuclear bar region, the intensity distributions of $^{12}$CO(1-0) and $^{13}$CO(1-0) emission trace moderate densities, and differ from the dense gas distributions sampled in C$^{18}$O(1-0), HCN(1-0), and HCO$^+$(1-0). We observe a constant HCN(1-0)-to-HCO$^+$(1-0) ratio of 1.2$\pm$0.1 across the whole $\sim$1 kpc bar. This indicates that HCN(1-0) and HCO$^+$(1-0) lines have intermediate optical depth, and that the corresponding $n_{\textrm{H}_2}$ of the gas producing the emission is of 10$^{4.5-6}$ cm$^{-3}$. We show that HCO$^+$(1-0) is thermalized and HCN(1-0) is close to thermalization. The very tight correlation between HCN(1-0) and HCO$^+$(1-0) intensities across the 1~kpc bar suggests that this ratio is more sensitive to the relative abundance of the two species than to the gas density. We confirm the angular offset ($\sim$10$^{\prime\prime}$) between the spatial distribution of molecular gas and the star formation sites. Finally, we find a breakdown of the $L_\textrm{IR}$-$L_\textrm{HCN}$ correlation at high spatial resolution due to the effect of incomplete sampling of star-forming regions by HCN emission in IC 342. The scatter of the $L_\textrm{IR}$-$L_\textrm{HCN}$ relation decreases as the spatial scale increases from 10$^{\prime\prime}$~to 30$^{\prime\prime}$ (170-510~pc), and is comparable to the scatter of the global relation at the scale of 340 pc.

astro-ph.GA↗

arXiv2023

Multimodal Large Language Model for Visual Navigation

Recent efforts to enable visual navigation using large language models have mainly focused on developing complex prompt systems. These systems incorporate instructions, observations, and history into massive text prompts, which are then combined with pre-trained large language models to facilitate visual navigation. In contrast, our approach aims to fine-tune large language models for visual navigation without extensive prompt engineering. Our design involves a simple text prompt, current observations, and a history collector model that gathers information from previous observations as input. For output, our design provides a probability distribution of possible actions that the agent can take during navigation. We train our model using human demonstrations and collision signals from the Habitat-Matterport 3D Dataset (HM3D). Experimental results demonstrate that our method outperforms state-of-the-art behavior cloning methods and effectively reduces collision rates.

cs.CV↗