arXiv subjects

Yun Liu

Publications and source records attributed to Yun Liu.

At least 127 records · Page 7Linked to original sources

arXiv2024

CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement

Understanding how humans cooperatively rearrange household objects is critical for VR/AR and human-robot interaction. However, in-depth studies on modeling these behaviors are under-researched due to the lack of relevant datasets. We fill this gap by presenting CORE4D, a novel large-scale 4D human-object-human interaction dataset focusing on collaborative object rearrangement, which encompasses diverse compositions of various object geometries, collaboration modes, and 3D scenes. With 1K human-object-human motion sequences captured in the real world, we enrich CORE4D by contributing an iterative collaboration retargeting strategy to augment motions to a variety of novel objects. Leveraging this approach, CORE4D comprises a total of 11K collaboration sequences spanning 3K real and virtual object shapes. Benefiting from extensive motion patterns provided by CORE4D, we benchmark two tasks aiming at generating human-object interaction: human-object motion forecasting and interaction synthesis. Extensive experiments demonstrate the effectiveness of our collaboration retargeting strategy and indicate that CORE4D has posed new challenges to existing human-object interaction generation methodologies.

cs.CV↗

arXiv2024

Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimicking

Learning generic skills for humanoid robots interacting with 3D scenes by mimicking human data is a key research challenge with significant implications for robotics and real-world applications. However, existing methodologies and benchmarks are constrained by the use of small-scale, manually collected demonstrations, lacking the general dataset and benchmark support necessary to explore scene geometry generalization effectively. To address this gap, we introduce Mimicking-Bench, the first comprehensive benchmark designed for generalizable humanoid-scene interaction learning through mimicking large-scale human animation references. Mimicking-Bench includes six household full-body humanoid-scene interaction tasks, covering 11K diverse object shapes, along with 20K synthetic and 3K real-world human interaction skill references. We construct a complete humanoid skill learning pipeline and benchmark approaches for motion retargeting, motion tracking, imitation learning, and their various combinations. Extensive experiments highlight the value of human mimicking for skill learning, revealing key challenges and research directions.

cs.RO↗

arXiv2024

ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping

In this paper, we introduce ManiVideo, a novel method for generating consistent and temporally coherent bimanual hand-object manipulation videos from given motion sequences of hands and objects. The core idea of ManiVideo is the construction of a multi-layer occlusion (MLO) representation that learns 3D occlusion relationships from occlusion-free normal maps and occlusion confidence maps. By embedding the MLO structure into the UNet in two forms, the model enhances the 3D consistency of dexterous hand-object manipulation. To further achieve the generalizable grasping of objects, we integrate Objaverse, a large-scale 3D object dataset, to address the scarcity of video data, thereby facilitating the learning of extensive object consistency. Additionally, we propose an innovative training strategy that effectively integrates multiple datasets, supporting downstream tasks such as human-centric hand-object manipulation video generation. Through extensive experiments, we demonstrate that our approach not only achieves video generation with plausible hand-object interaction and generalizable objects, but also outperforms existing SOTA methods.

cs.CV↗

arXiv2024

Libri2Vox Dataset: Target Speaker Extraction with Diverse Speaker Conditions and Synthetic Data

Target speaker extraction (TSE) is essential in speech processing applications, particularly in scenarios with complex acoustic environments. Current TSE systems face challenges in limited data diversity and a lack of robustness in real-world conditions, primarily because they are trained on artificially mixed datasets with limited speaker variability and unrealistic noise profiles. To address these challenges, we propose Libri2Vox, a new dataset that combines clean target speech from the LibriTTS dataset with interference speech from the noisy VoxCeleb2 dataset, providing a large and diverse set of speakers under realistic noisy conditions. We also augment Libri2Vox with synthetic speakers generated using state-of-the-art speech generative models to enhance speaker diversity. Additionally, to further improve the effectiveness of incorporating synthetic data, curriculum learning is implemented to progressively train TSE models with increasing levels of difficulty. Extensive experiments across multiple TSE architectures reveal varying degrees of improvement, with SpeakerBeam demonstrating the most substantial gains: a 1.39 dB improvement in signal-to-distortion ratio (SDR) on the Libri2Talker test set compared to baseline training. Building upon these results, we further enhanced performance through our speaker similarity-based curriculum learning approach with the Conformer architecture, achieving an additional 0.78 dB improvement over conventional random sampling methods in which data samples are randomly selected from the entire dataset. These results demonstrate the complementary benefits of diverse real-world data, synthetic speaker augmentation, and structured training strategies in building robust TSE systems.

cs.SD↗

arXiv2024

Towards Open-Vocabulary Video Semantic Segmentation

Semantic segmentation in videos has been a focal point of recent research. However, existing models encounter challenges when faced with unfamiliar categories. To address this, we introduce the Open Vocabulary Video Semantic Segmentation (OV-VSS) task, designed to accurately segment every pixel across a wide range of open-vocabulary categories, including those that are novel or previously unexplored. To enhance OV-VSS performance, we propose a robust baseline, OV2VSS, which integrates a spatial-temporal fusion module, allowing the model to utilize temporal relationships across consecutive frames. Additionally, we incorporate a random frame enhancement module, broadening the model's understanding of semantic context throughout the entire video sequence. Our approach also includes video text encoding, which strengthens the model's capability to interpret textual information within the video context. Comprehensive evaluations on benchmark datasets such as VSPW and Cityscapes highlight OV-VSS's zero-shot generalization capabilities, especially in handling novel categories. The results validate OV2VSS's effectiveness, demonstrating improved performance in semantic segmentation tasks across diverse video datasets.

cs.MM↗

arXiv2024

Momentum-Accelerated Richardson(m) and Their Multilevel Neural Solvers

Recently, designing neural solvers for large-scale linear systems of equations has emerged as a promising approach in scientific and engineering computing. This paper first introduce the Richardson(m) neural solver by employing a meta network to predict the weights of the long-step Richardson iterative method. Next, by incorporating momentum and preconditioning techniques, we further enhance convergence. Numerical experiments on anisotropic second-order elliptic equations demonstrate that these new solvers achieve faster convergence and lower computational complexity compared to both the Chebyshev iterative method with optimal weights and the Chebyshev semi-iteration method. To address the strong dependence of the aforementioned single-level neural solvers on PDE parameters and grid size, we integrate them with two multilevel neural solvers developed in recent years. Using alternating optimization techniques, we construct Richardson(m)-FNS for anisotropic equations and NAG-Richardson(m)-WANS for the Helmholtz equation. Numerical experiments show that these two multilevel neural solvers effectively overcome the drawback of single-level methods, providing better robustness and computational efficiency.

math.NA↗

arXiv2024

Exploring Frequency-Inspired Optimization in Transformer for Efficient Single Image Super-Resolution

Transformer-based methods have exhibited remarkable potential in single image super-resolution (SISR) by effectively extracting long-range dependencies. However, most of the current research in this area has prioritized the design of transformer blocks to capture global information, while overlooking the importance of incorporating high-frequency priors, which we believe could be beneficial. In our study, we conducted a series of experiments and found that transformer structures are more adept at capturing low-frequency information, but have limited capacity in constructing high-frequency representations when compared to their convolutional counterparts. Our proposed solution, the cross-refinement adaptive feature modulation transformer (CRAFT), integrates the strengths of both convolutional and transformer structures. It comprises three key components: the high-frequency enhancement residual block (HFERB) for extracting high-frequency information, the shift rectangle window attention block (SRWAB) for capturing global information, and the hybrid fusion block (HFB) for refining the global representation. To tackle the inherent intricacies of transformer structures, we introduce a frequency-guided post-training quantization (PTQ) method aimed at enhancing CRAFT's efficiency. These strategies incorporate adaptive dual clipping and boundary refinement. To further amplify the versatility of our proposed approach, we extend our PTQ strategy to function as a general quantization method for transformer-based SISR techniques. Our experimental findings showcase CRAFT's superiority over current state-of-the-art methods, both in full-precision and quantization scenarios. These results underscore the efficacy and universality of our PTQ strategy. The source code is available at: https://github.com/AVC2-UESTC/Frequency-Inspired-Optimization-for-EfficientSR.git.

cs.CV↗

arXiv2024

Health AI Developer Foundations

Robust medical Machine Learning (ML) models have the potential to revolutionize healthcare by accelerating clinical research, improving workflows and outcomes, and producing novel insights or capabilities. Developing such ML models from scratch is cost prohibitive and requires substantial compute, data, and time (e.g., expert labeling). To address these challenges, we introduce Health AI Developer Foundations (HAI-DEF), a suite of pre-trained, domain-specific foundation models, tools, and recipes to accelerate building ML for health applications. The models cover various modalities and domains, including radiology (X-rays and computed tomography), histopathology, dermatological imaging, and audio. These models provide domain specific embeddings that facilitate AI development with less labeled data, shorter training times, and reduced computational costs compared to traditional approaches. In addition, we utilize a common interface and style across these models, and prioritize usability to enable developers to integrate HAI-DEF efficiently. We present model evaluations across various tasks and conclude with a discussion of their application and evaluation, covering the importance of ensuring efficacy, fairness, and equity. Finally, while HAI-DEF and specifically the foundation models lower the barrier to entry for ML in healthcare, we emphasize the importance of validation with problem- and population-specific data for each desired usage setting. This technical report will be updated over time as more modalities and features are added.

cs.LG↗

arXiv2024

PointCG: Self-supervised Point Cloud Learning via Joint Completion and Generation

The core of self-supervised point cloud learning lies in setting up appropriate pretext tasks, to construct a pre-training framework that enables the encoder to perceive 3D objects effectively. In this paper, we integrate two prevalent methods, masked point modeling (MPM) and 3D-to-2D generation, as pretext tasks within a pre-training framework. We leverage the spatial awareness and precise supervision offered by these two methods to address their respective limitations: ambiguous supervision signals and insensitivity to geometric information. Specifically, the proposed framework, abbreviated as PointCG, consists of a Hidden Point Completion (HPC) module and an Arbitrary-view Image Generation (AIG) module. We first capture visible points from arbitrary views as inputs by removing hidden points. Then, HPC extracts representations of the inputs with an encoder and completes the entire shape with a decoder, while AIG is used to generate rendered images based on the visible points' representations. Extensive experiments demonstrate the superiority of the proposed method over the baselines in various downstream tasks. Our code will be made available upon acceptance.

cs.CV↗

arXiv2024

Cosmological Prediction of the Void and Galaxy Clustering Measurements in the CSST Spectroscopic Survey

The void power spectrum is related to the clustering of low-density regions in the large-scale structure (LSS) of the Universe, and can be used as an effective cosmological probe to extract the information of the LSS. We generate the galaxy mock catalogs from Jiutian simulation, and identify voids using the watershed algorithm for studying the cosmological constraint strength of the China Space Station Telescope (CSST) spectroscopic survey. The galaxy and void auto power spectra and void-galaxy cross power spectra at $z=0.3$, 0.6, and 0.9 are derived from the mock catalogs. To fit the full power spectra, we propose to use the void average effective radius at a given redshift to simplify the theoretical model, and adopt the Markov Chain Monte Carlo (MCMC) technique to implement the constraints on the cosmological and void parameters. The systematical parameters, such as galaxy and void biases, and noise terms in the power spectra are also included in the fitting process. We find that our theoretical model can correctly extract the cosmological information from the galaxy and void power spectra, which demonstrates its feasibility and effectivity. The joint constraint accuracy of the cosmological parameters can be improved by $\sim20\%$ compared to that from the galaxy power spectrum only. The fitting results of the void density profile and systematical parameters are also well constrained and consistent with the expectation. This indicates that the void clustering measurement can be an effective complement to the galaxy clustering probe, especially for the next generation galaxy surveys.

astro-ph.CO↗

arXiv2024

Scaling Wearable Foundation Models

Wearable sensors have become ubiquitous thanks to a variety of health tracking features. The resulting continuous and longitudinal measurements from everyday life generate large volumes of data; however, making sense of these observations for scientific and actionable insights is non-trivial. Inspired by the empirical success of generative modeling, where large neural networks learn powerful representations from vast amounts of text, image, video, or audio data, we investigate the scaling properties of sensor foundation models across compute, data, and model size. Using a dataset of up to 40 million hours of in-situ heart rate, heart rate variability, electrodermal activity, accelerometer, skin temperature, and altimeter per-minute data from over 165,000 people, we create LSM, a multimodal foundation model built on the largest wearable-signals dataset with the most extensive range of sensor modalities to date. Our results establish the scaling laws of LSM for tasks such as imputation, interpolation and extrapolation, both across time and sensor modalities. Moreover, we highlight how LSM enables sample-efficient downstream learning for tasks like exercise and activity recognition.

cs.LG↗

arXiv2024

Improving curriculum learning for target speaker extraction with synthetic speakers

Target speaker extraction (TSE) aims to isolate individual speaker voices from complex speech environments. The effectiveness of TSE systems is often compromised when the speaker characteristics are similar to each other. Recent research has introduced curriculum learning (CL), in which TSE models are trained incrementally on speech samples of increasing complexity. In CL training, the model is first trained on samples with low speaker similarity between the target and interference speakers, and then on samples with high speaker similarity. To further improve CL, this paper uses a $k$-nearest neighbor-based voice conversion method to simulate and generate speech of diverse interference speakers, and then uses the generated data as part of the CL. Experiments demonstrate that training data based on synthetic speakers can effectively enhance the model's capabilities and significantly improve the performance of multiple TSE systems.

cs.SD↗

arXiv2024

A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models

Large language models (LLMs) hold promise to serve complex health information needs but also have the potential to introduce harm and exacerbate health disparities. Reliably evaluating equity-related model failures is a critical step toward developing systems that promote health equity. We present resources and methodologies for surfacing biases with potential to precipitate equity-related harms in long-form, LLM-generated answers to medical questions and conduct a large-scale empirical case study with the Med-PaLM 2 LLM. Our contributions include a multifactorial framework for human assessment of LLM-generated answers for biases, and EquityMedQA, a collection of seven datasets enriched for adversarial queries. Both our human assessment framework and dataset design process are grounded in an iterative participatory approach and review of Med-PaLM 2 answers. Through our empirical study, we find that our approach surfaces biases that may be missed via narrower evaluation approaches. Our experience underscores the importance of using diverse assessment methodologies and involving raters of varying backgrounds and expertise. While our approach is not sufficient to holistically assess whether the deployment of an AI system promotes equitable health outcomes, we hope that it can be leveraged and built upon towards a shared goal of LLMs that promote accessible and equitable healthcare.

cs.CY↗

arXiv2024

Resolving the Valence of Iron Oxides by Resonant Photoemission Spectroscopy

Precisely determining the oxidation states of metal cations within variable-valence transition metal oxides remains a significant challenge, yet it is crucial for understanding and predicting the properties of these technologically important materials. Iron oxides, in particular, exhibit a remarkable diversity of electronic structures due to the variable valence states of iron (Fe2+ and Fe3+), however, quantitative analysis using conventional X-ray photoelectron spectroscopy (XPS) is challenging because of significant overlapping of the Fe2p spectra among different oxidation states. In this study, we leverage the intriguing case of Pt supported FeO2 phase of monolayer thickness (ML) as a model system and employ Resonant Photoemission Spectroscopy (ResPES) to directly quantify the cation valence states and compositional ratios in this complex Fe oxide. Our results reveal that this ultrathin FeO2 film (Pt-O-Fe-O), contrary to the +3 valence predicted by density functional theory (DFT), consists of an equal mixture of Fe2+ and Fe3+ cations, yielding an average valence of +2.5. Structurally, FeO2 is likely derived from the Fe3O4 sublattice, featuring an octahedral Fe layer (50% Fe3+ and 50% Fe2+) bonded to upper and lower oxygen layers.

cond-mat.mtrl-sci↗

arXiv2024

Void Number Counts as a Cosmological Probe for the Large-Scale Structure

Void number counts (VNC) indicates the number of low-density regions in the large-scale structure (LSS) of the Universe, and we propose to use it as an effective cosmological probe. By generating the galaxy mock catalog based on Jiutian simulations and considering the spectroscopic survey strategy and instrumental design of the China Space Station Telescope (CSST), which can reach a magnitude limit $\sim$23 AB mag and spectral resolution $R\gtrsim200$ with a sky coverage 17,500 deg$^2$, we identify voids using the watershed algorithm without any assumption of void shape, and obtain the mock void catalog and data of the VNC in six redshift bins from $z=0.3$ to1.3. We use the Markov Chain Monte Carlo (MCMC) method to constrain the cosmological and VNC parameters. The void linear underdensity threshold $δ_{\rm v}$ in the theoretical model is set to be a free parameter at a given redshift to fit the VNC data and explore its redshift evolution. We find that, the VNC can correctly derive the cosmological information, and the constraint strength on the cosmological parameters is comparable to that from the void size function (VSF) method, which can reach a few percentage levels in the CSST full spectroscopic survey. This is because that, since the VNC is not sensitive to void shape, the modified theoretical model can match the data better by integrating over void features, and more voids could be included in the VNC analysis by applying simpler selection criteria, which will improve the statistical significance. It indicates that the VNC can be an effective cosmological probe for exploring the LSS.

astro-ph.CO↗

arXiv2024

Generalized Orthogonal Chirp Division Multiplexing in Doubly Selective Channels

In recent years, orthogonal chirp division modulation (OCDM) has gained attention as a robust communication waveform due to its strong resistance to both time-domain and frequency-domain interference. However, similar to orthogonal frequency division multiplexing (OFDM), OCDM suffers from a high peak-to-average power ratio (PAPR), resulting in increased hardware costs and reduced energy efficiency of the transmitter's power amplifiers. In this work, we introduce a novel unitary transform called the Generalized Discrete Fresnel Transform (GDFnT) and propose a new waveform based on this transform, named Generalized Orthogonal Chirp Division Modulation (GOCDM). In GOCDM, data symbols from the constellation diagram are independently placed in the Generalized Fresnel (GF) domain. We derive the GF-domain channel matrix for the GOCDM system under time-frequency doubly selective channels and leverages the sparsity of the GF-domain channel matrix to design an iterative receiver based on the message-passing algorithm. Simulation results demonstrate that GOCDM achieves better PAPR performance than OCDM without compromising bit error rate (BER) performance.

cs.IT↗

arXiv2024

TCR-GPT: Integrating Autoregressive Model and Reinforcement Learning for T-Cell Receptor Repertoires Generation

T-cell receptors (TCRs) play a crucial role in the immune system by recognizing and binding to specific antigens presented by infected or cancerous cells. Understanding the sequence patterns of TCRs is essential for developing targeted immune therapies and designing effective vaccines. Language models, such as auto-regressive transformers, offer a powerful solution to this problem by learning the probability distributions of TCR repertoires, enabling the generation of new TCR sequences that inherit the underlying patterns of the repertoire. We introduce TCR-GPT, a probabilistic model built on a decoder-only transformer architecture, designed to uncover and replicate sequence patterns in TCR repertoires. TCR-GPT demonstrates an accuracy of 0.953 in inferring sequence probability distributions measured by Pearson correlation coefficient. Furthermore, by leveraging Reinforcement Learning(RL), we adapted the distribution of TCR sequences to generate TCRs capable of recognizing specific peptides, offering significant potential for advancing targeted immune therapies and vaccine development. With the efficacy of RL, fine-tuned pretrained TCR-GPT models demonstrated the ability to produce TCR repertoires likely to bind specific peptides, illustrating RL's efficiency in enhancing the model's adaptability to the probability distributions of biologically relevant TCR sequences.

cs.LG↗

arXiv2024

Haze-Aware Attention Network for Single-Image Dehazing

Single-image dehazing is a pivotal challenge in computer vision that seeks to remove haze from images and restore clean background details. Recognizing the limitations of traditional physical model-based methods and the inefficiencies of current attention-based solutions, we propose a new dehazing network combining an innovative Haze-Aware Attention Module (HAAM) with a Multiscale Frequency Enhancement Module (MFEM). The HAAM is inspired by the atmospheric scattering model, thus skillfully integrating physical principles into high-dimensional features for targeted dehazing. It picks up on latent features during the image restoration process, which gives a significant boost to the metrics, while the MFEM efficiently enhances high-frequency details, thus sidestepping wavelet or Fourier transform complexities. It employs multiscale fields to extract and emphasize key frequency components with minimal parameter overhead. Integrated into a simple U-Net framework, our Haze-Aware Attention Network (HAA-Net) for single-image dehazing significantly outperforms existing attention-based and transformer models in efficiency and effectiveness. Tested across various public datasets, the HAA-Net sets new performance benchmarks. Our work not only advances the field of image dehazing but also offers insights into the design of attention mechanisms for broader applications in computer vision.

cs.CV↗