arXiv subjects

Yi Xu

Publications and source records attributed to Yi Xu.

At least 217 records · Page 12Linked to original sources

arXiv2024

Uncertainty-Guided Enhancement on Driving Perception System via Foundation Models

Multimodal foundation models offer promising advancements for enhancing driving perception systems, but their high computational and financial costs pose challenges. We develop a method that leverages foundation models to refine predictions from existing driving perception models -- such as enhancing object classification accuracy -- while minimizing the frequency of using these resource-intensive models. The method quantitatively characterizes uncertainties in the perception model's predictions and engages the foundation model only when these uncertainties exceed a pre-specified threshold. Specifically, it characterizes uncertainty by calibrating the perception model's confidence scores into theoretical lower bounds on the probability of correct predictions using conformal prediction. Then, it sends images to the foundation model and queries for refining the predictions only if the theoretical bound of the perception model's outcome is below the threshold. Additionally, we propose a temporal inference mechanism that enhances prediction accuracy by integrating historical predictions, leading to tighter theoretical bounds. The method demonstrates a 10 to 15 percent improvement in prediction accuracy and reduces the number of queries to the foundation model by 50 percent, based on quantitative evaluations from driving datasets.

cs.CV↗

arXiv2024

Diffusion Models For Multi-Modal Generative Modeling

Diffusion-based generative modeling has been achieving state-of-the-art results on various generation tasks. Most diffusion models, however, are limited to a single-generation modeling. Can we generalize diffusion models with the ability of multi-modal generative training for more generalizable modeling? In this paper, we propose a principled way to define a diffusion model by constructing a unified multi-modal diffusion model in a common diffusion space. We define the forward diffusion process to be driven by an information aggregation from multiple types of task-data, e.g., images for a generation task and labels for a classification task. In the reverse process, we enforce information sharing by parameterizing a shared backbone denoising network with additional modality-specific decoder heads. Such a structure can simultaneously learn to generate different types of multi-modal data with a multi-task loss, which is derived from a new multi-modal variational lower bound that generalizes the standard diffusion model. We propose several multimodal generation settings to verify our framework, including image transition, masked-image training, joint image-label and joint image-representation generative modeling. Extensive experimental results on ImageNet indicate the effectiveness of our framework for various multi-modal generative modeling, which we believe is an important research direction worthy of more future explorations.

cs.CV↗

arXiv2024

Generative AI Is Not Ready for Clinical Use in Patient Education for Lower Back Pain Patients, Even With Retrieval-Augmented Generation

Low back pain (LBP) is a leading cause of disability globally. Following the onset of LBP and subsequent treatment, adequate patient education is crucial for improving functionality and long-term outcomes. Despite advancements in patient education strategies, significant gaps persist in delivering personalized, evidence-based information to patients with LBP. Recent advancements in large language models (LLMs) and generative artificial intelligence (GenAI) have demonstrated the potential to enhance patient education. However, their application and efficacy in delivering educational content to patients with LBP remain underexplored and warrant further investigation. In this study, we introduce a novel approach utilizing LLMs with Retrieval-Augmented Generation (RAG) and few-shot learning to generate tailored educational materials for patients with LBP. Physical therapists manually evaluated our model responses for redundancy, accuracy, and completeness using a Likert scale. In addition, the readability of the generated education materials is assessed using the Flesch Reading Ease score. The findings demonstrate that RAG-based LLMs outperform traditional LLMs, providing more accurate, complete, and readable patient education materials with less redundancy. Having said that, our analysis reveals that the generated materials are not yet ready for use in clinical practice. This study underscores the potential of AI-driven models utilizing RAG to improve patient education for LBP; however, significant challenges remain in ensuring the clinical relevance and granularity of content generated by these models.

cs.AI↗

arXiv2024

Good Idea or Not, Representation of LLM Could Tell

In the ever-expanding landscape of academic research, the proliferation of ideas presents a significant challenge for researchers: discerning valuable ideas from the less impactful ones. The ability to efficiently evaluate the potential of these ideas is crucial for the advancement of science and paper review. In this work, we focus on idea assessment, which aims to leverage the knowledge of large language models to assess the merit of scientific ideas. First, we investigate existing text evaluation research and define the problem of quantitative evaluation of ideas. Second, we curate and release a benchmark dataset from nearly four thousand manuscript papers with full texts, meticulously designed to train and evaluate the performance of different approaches to this task. Third, we establish a framework for quantifying the value of ideas by employing representations in a specific layer of large language models. Experimental results show that the scores predicted by our method are relatively consistent with those of humans. Our findings suggest that the representations of large language models hold more potential in quantifying the value of ideas than their generative outputs, demonstrating a promising avenue for automating the idea assessment process.

cs.CL↗

arXiv2024

Balmer Decrement Anomalies in Galaxies at z ~ 6 Found by JWST Observations: Density-Bounded Nebulae or Excited H I Clouds?

We investigate the physical origins of the Balmer decrement anomalies in GS-NDG-9422 (Cameron et al. 2023) and RXCJ2248-ID (Topping et al. 2024) galaxies at $z\sim 6$ whose $\mathrm{H}α/\mathrm{H}β$ values are significantly smaller than $2.7$, the latter of which also shows anomalous $\mathrm{H}γ/\mathrm{H}β$ and $\mathrm{H}δ/\mathrm{H}β$ values beyond the errors. Because the anomalous Balmer decrements are not reproduced under the Case B recombination, we explore the nebulae with the optical depths smaller and larger than the Case B recombination by physical modeling. We find two cases quantitatively explaining the anomalies; 1) density-bounded nebulae that are opaque only up to around Ly$γ$-Ly8 transitions and 2) ionization-bounded nebulae partly/fully surrounded by optically-thick excited H{\sc i} clouds. The case of 1) produces more H$β$ photons via Ly$γ$ absorption in the nebulae, requiring fine tuning in optical depth values, while this case helps ionizing photon escape for cosmic reionization. The case of 2) needs the optically-thick excited H{\sc i} clouds with $N_2\simeq 10^{12}-10^{13}$ $\mathrm{cm^{-2}}$, where $N_2$ is the column density of the hydrogen atom with the principal quantum number of $n=2$. Interestingly, the high $N_2$ values qualitatively agree with the recent claims for GS-NDG-9422 with the strong nebular continuum requiring a number of $2s$-state electrons and for RXCJ2248-ID with the dense ionized regions likely coexisting with the optically-thick clouds. While the physical origin of the optically-thick excited H{\sc i} clouds is unclear, these results may suggest gas clouds with excessive collisional excitation caused by an amount of accretion and supernovae in the high-$z$ galaxies.

astro-ph.GA↗

arXiv2024

Strong He I Emission Lines in High N/O Galaxies at $z \sim 6$ Identified in JWST Spectra: High He/H Abundance Ratios or High Electron Densities?

We present HeI/H$β$-flux and He/H-abundance ratios in three JWST galaxies with significant constraints on N/O-abundance ratios, GS-NDG-9422, RXCJ2248-ID, and GLASS150008 at $z\sim 6$ mostly with the spectroscopic coverage from HeI$λ$4471 and HeII$λ$4686 to HeI$λ$7065, comparing with 68 local-dwarf galaxies. We find that these high-$z$ galaxies present strong HeI emission with HeI/H$β$ flux ratios generally larger than those of local-dwarf galaxies. We derive He/H with all of the detected HeI, HeII, and $2-3$ hydrogen Balmer lines in the same manner as the local He/H determination conducted for cosmology studies. These high-$z$ galaxies show He overabundance He/H$\gtrsim 0.10$ or high electron density $n_\mathrm{e}\sim 10^{3-4}$ cm$^{-3}$ much larger than local values at low O/H, $12+\log \mathrm{(O/H)}=7-8$. In contrast, we obtain low He/H and $n_\mathrm{e}$ values for our local-dwarf galaxies by the same technique with the same helium and hydrogen lines, and confirm that the difference between the high-$z$ and local-dwarf galaxies are not mimicked by systematics. While two scenarios of 1) He overabundance and 2) high electron density are not clearly concluded, we find that there is a positive correlation on the He/H-N/O or $n_\mathrm{e}$-N/O plane by the comparison of the high-$z$ and local-dwarf galaxies. The scenario 1) suggests that the overabundant helium and nitrogen are not explained by the standard chemical enrichment of core-collapse supernovae, but the CNO-cycle products and equilibrium ratios, respectively. The scenario 2) indicates that the strong helium lines are originated from the central dense clouds of the high-$z$ galaxies by excessive collisional excitation.

astro-ph.GA↗

arXiv2024

Skeleton-Based Human Action Recognition with Noisy Labels

Understanding human actions from body poses is critical for assistive robots sharing space with humans in order to make informed and safe decisions about the next interaction. However, precise temporal localization and annotation of activity sequences is time-consuming and the resulting labels are often noisy. If not effectively addressed, label noise negatively affects the model's training, resulting in lower recognition quality. Despite its importance, addressing label noise for skeleton-based action recognition has been overlooked so far. In this study, we bridge this gap by implementing a framework that augments well-established skeleton-based human action recognition methods with label-denoising strategies from various research areas to serve as the initial benchmark. Observations reveal that these baselines yield only marginal performance when dealing with sparse skeleton data. Consequently, we introduce a novel methodology, NoiseEraSAR, which integrates global sample selection, co-teaching, and Cross-Modal Mixture-of-Experts (CM-MOE) strategies, aimed at mitigating the adverse impacts of label noise. Our proposed approach demonstrates better performance on the established benchmark, setting new state-of-the-art standards. The source code for this study is accessible at https://github.com/xuyizdby/NoiseEraSAR.

cs.CV↗

arXiv2024

PanoFree: Tuning-Free Holistic Multi-view Image Generation with Cross-view Self-Guidance

Immersive scene generation, notably panorama creation, benefits significantly from the adaptation of large pre-trained text-to-image (T2I) models for multi-view image generation. Due to the high cost of acquiring multi-view images, tuning-free generation is preferred. However, existing methods are either limited to simple correspondences or require extensive fine-tuning to capture complex ones. We present PanoFree, a novel method for tuning-free multi-view image generation that supports an extensive array of correspondences. PanoFree sequentially generates multi-view images using iterative warping and inpainting, addressing the key issues of inconsistency and artifacts from error accumulation without the need for fine-tuning. It improves error accumulation by enhancing cross-view awareness and refines the warping and inpainting processes via cross-view guidance, risky area estimation and erasing, and symmetric bidirectional guided generation for loop closure, alongside guidance-based semantic and density control for scene structure preservation. In experiments on Planar, 360°, and Full Spherical Panoramas, PanoFree demonstrates significant error reduction, improves global consistency, and boosts image quality without extra fine-tuning. Compared to existing methods, PanoFree is up to 5x more efficient in time and 3x more efficient in GPU memory usage, and maintains superior diversity of results (2x better in our user study). PanoFree offers a viable alternative to costly fine-tuning or the use of additional pre-trained models. Project website at https://panofree.github.io/.

cs.CV↗

arXiv2024

ParCo: Part-Coordinating Text-to-Motion Synthesis

We study a challenging task: text-to-motion synthesis, aiming to generate motions that align with textual descriptions and exhibit coordinated movements. Currently, the part-based methods introduce part partition into the motion synthesis process to achieve finer-grained generation. However, these methods encounter challenges such as the lack of coordination between different part motions and difficulties for networks to understand part concepts. Moreover, introducing finer-grained part concepts poses computational complexity challenges. In this paper, we propose Part-Coordinating Text-to-Motion Synthesis (ParCo), endowed with enhanced capabilities for understanding part motions and communication among different part motion generators, ensuring a coordinated and fined-grained motion synthesis. Specifically, we discretize whole-body motion into multiple part motions to establish the prior concept of different parts. Afterward, we employ multiple lightweight generators designed to synthesize different part motions and coordinate them through our part coordination module. Our approach demonstrates superior performance on common benchmarks with economic computations, including HumanML3D and KIT-ML, providing substantial evidence of its effectiveness. Code is available at https://github.com/qrzou/ParCo .

cs.CV↗

arXiv2024

Robust Multi-Task Learning with Excess Risks

Multi-task learning (MTL) considers learning a joint model for multiple tasks by optimizing a convex combination of all task losses. To solve the optimization problem, existing methods use an adaptive weight updating scheme, where task weights are dynamically adjusted based on their respective losses to prioritize difficult tasks. However, these algorithms face a great challenge whenever label noise is present, in which case excessive weights tend to be assigned to noisy tasks that have relatively large Bayes optimal errors, thereby overshadowing other tasks and causing performance to drop across the board. To overcome this limitation, we propose Multi-Task Learning with Excess Risks (ExcessMTL), an excess risk-based task balancing method that updates the task weights by their distances to convergence instead. Intuitively, ExcessMTL assigns higher weights to worse-trained tasks that are further from convergence. To estimate the excess risks, we develop an efficient and accurate method with Taylor approximation. Theoretically, we show that our proposed algorithm achieves convergence guarantees and Pareto stationarity. Empirically, we evaluate our algorithm on various MTL benchmarks and demonstrate its superior performance over existing methods in the presence of label noise. Our code is available at https://github.com/yifei-he/ExcessMTL.

cs.LG↗

arXiv2024

CXL Shared Memory Programming: Barely Distributed and Almost Persistent

While Compute Express Link (CXL) enables support for cache-coherent shared memory among multiple nodes, it also introduces new types of failures--processes can fail before data does, or data might fail before a process does. The lack of a failure model for CXL-based shared memory makes it challenging to understand and mitigate these failures. To solve these challenges, in this paper, we describe a model categorizing and handling the CXL-based shared memory's failures: data and process failures. Data failures in CXL-based shared memory render data inaccessible or inconsistent for a currently running application. We argue that such failures are unlike data failures in distributed storage systems and require CXL-specific handling. To address this, we look into traditional data failure mitigation techniques like erasure coding and replication and propose new solutions to better handle data failures in CXL-based shared memory systems. Next, we look into process failures and compare the failures and potential solutions with PMEM's failure model and programming solutions. We argue that although PMEM shares some of CXL's characteristics, it does not fully address CXL's volatile nature and low access latencies. Finally, taking inspiration from PMEM programming solutions, we propose techniques to handle these new failures. Thus, this paper is the first work to define the CXL-based shared memory failure model and propose tailored solutions that address challenges specific to CXL-based systems.

cs.DC↗

arXiv2024

Efficient Personalized Text-to-image Generation by Leveraging Textual Subspace

Personalized text-to-image generation has attracted unprecedented attention in the recent few years due to its unique capability of generating highly-personalized images via using the input concept dataset and novel textual prompt. However, previous methods solely focus on the performance of the reconstruction task, degrading its ability to combine with different textual prompt. Besides, optimizing in the high-dimensional embedding space usually leads to unnecessary time-consuming training process and slow convergence. To address these issues, we propose an efficient method to explore the target embedding in a textual subspace, drawing inspiration from the self-expressiveness property. Additionally, we propose an efficient selection strategy for determining the basis vectors of the textual subspace. The experimental evaluations demonstrate that the learned embedding can not only faithfully reconstruct input image, but also significantly improves its alignment with novel input textual prompt. Furthermore, we observe that optimizing in the textual subspace leads to an significant improvement of the robustness to the initial word, relaxing the constraint that requires users to input the most relevant initial word. Our method opens the door to more efficient representation learning for personalized text-to-image generation.

cs.AI↗

arXiv2024

The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge

This report introduces an enhanced method for the Foundational Few-Shot Object Detection (FSOD) task, leveraging the vision-language model (VLM) for object detection. However, on specific datasets, VLM may encounter the problem where the detected targets are misaligned with the target concepts of interest. This misalignment hinders the zero-shot performance of VLM and the application of fine-tuning methods based on pseudo-labels. To address this issue, we propose the VLM+ framework, which integrates the multimodal large language model (MM-LLM). Specifically, we use MM-LLM to generate a series of referential expressions for each category. Based on the VLM predictions and the given annotations, we select the best referential expression for each category by matching the maximum IoU. Subsequently, we use these referential expressions to generate pseudo-labels for all images in the training set and then combine them with the original labeled data to fine-tune the VLM. Additionally, we employ iterative pseudo-label generation and optimization to further enhance the performance of the VLM. Our approach achieve 32.56 mAP in the final test.

cs.CV↗

arXiv2024

OrientDream: Streamlining Text-to-3D Generation with Explicit Orientation Control

In the evolving landscape of text-to-3D technology, Dreamfusion has showcased its proficiency by utilizing Score Distillation Sampling (SDS) to optimize implicit representations such as NeRF. This process is achieved through the distillation of pretrained large-scale text-to-image diffusion models. However, Dreamfusion encounters fidelity and efficiency constraints: it faces the multi-head Janus issue and exhibits a relatively slow optimization process. To circumvent these challenges, we introduce OrientDream, a camera orientation conditioned framework designed for efficient and multi-view consistent 3D generation from textual prompts. Our strategy emphasizes the implementation of an explicit camera orientation conditioned feature in the pre-training of a 2D text-to-image diffusion module. This feature effectively utilizes data from MVImgNet, an extensive external multi-view dataset, to refine and bolster its functionality. Subsequently, we utilize the pre-conditioned 2D images as a basis for optimizing a randomly initialized implicit representation (NeRF). This process is significantly expedited by a decoupled back-propagation technique, allowing for multiple updates of implicit parameters per optimization cycle. Our experiments reveal that our method not only produces high-quality NeRF models with consistent multi-view properties but also achieves an optimization speed significantly greater than existing methods, as quantified by comparative metrics.

cs.CV↗

arXiv2024

JWST Measurements of Neutral Hydrogen Fractions and Ionized Bubble Sizes at $z=7-12$ Obtained with Ly$α$ Damping Wing Absorptions in 27 Bright Continuum Galaxies

We present volume-averaged neutral hydrogen fractions $x_{\rm \HI}$ and ionized bubble radii $R_{\rm b}$ measured with Ly$α$ damping wing absorption of galaxies at the epoch of reionization. We combine JWST/NIRSpec spectra taken by CEERS, GO-1433, DDT-2750, and JADES programs, and obtain a sample containing 27 bright UV-continuum ($M_{\rm UV}<-18.5~{\rm mag}$) galaxies at $7<z<12$. We construct 4 composite spectra binned by redshift, and find the clear evolution of softening break towards high redshift at the rest-frame $1216$ Å, suggesting the increase of Ly$α$ damping wing absorption. We estimate Ly$α$ damping wing absorption in the galaxy spectra with realistic templates including Ly$α$ emission and circum-galactic medium absorptions. Assuming the standard inside-out reionization picture having an ionized bubble with radius $R_b$ around a galaxy embedded in the intergalactic medium with $x_{\rm \HI}$, we obtain $x_{\rm \HI}$ ($R_{\rm b}$) values generally increasing (decreasing) from $x_{\rm \HI}={0.53}^{+0.18}_{-0.47}$ to ${0.92}^{+0.08}_{-0.10}$ ($\log R_{\rm b}={1.67}^{+0.14}_{-0.16}$ to ${-0.69}^{+0.89}_{-0.24}$ comoving Mpc) at redshift $7.12^{+0.06}_{-0.08}$ to $9.91^{+1.49}_{-1.15}$. The redshift evolution of $x_{\rm \HI}$ indicates a moderately late reionization history consistent with the one previously suggested from the electron scattering of cosmic microwave background and the evolution of UV luminosity function with an escape fraction $f_{\rm esc}\sim 0.2$. Our ${R_{\rm b}}$ measurements suggest that bubble sizes could be up to a few dex larger than the cosmic average values estimated by analytic calculations for a given $x_{\rm \HI}$, while our $R_{\rm b}$ measurements are roughly comparable with the values for merged ionized bubbles around bright galaxies predicted by recent numerical simulations.

astro-ph.GA↗

arXiv2024

PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo

We present a novel framework named PlaneMVS for 3D plane reconstruction from multiple input views with known camera poses. Most previous learning-based plane reconstruction methods reconstruct 3D planes from single images, which highly rely on single-view regression and suffer from depth scale ambiguity. In contrast, we reconstruct 3D planes with a multi-view-stereo (MVS) pipeline that takes advantage of multi-view geometry. We decouple plane reconstruction into a semantic plane detection branch and a plane MVS branch. The semantic plane detection branch is based on a single-view plane detection framework but with differences. The plane MVS branch adopts a set of slanted plane hypotheses to replace conventional depth hypotheses to perform plane sweeping strategy and finally learns pixel-level plane parameters and its planar depth map. We present how the two branches are learned in a balanced way, and propose a soft-pooling loss to associate the outputs of the two branches and make them benefit from each other. Extensive experiments on various indoor datasets show that PlaneMVS significantly outperforms state-of-the-art (SOTA) single-view plane reconstruction methods on both plane detection and 3D geometry metrics. Our method even outperforms a set of SOTA learning-based MVS methods thanks to the learned plane priors. To the best of our knowledge, this is the first work on 3D plane reconstruction within an end-to-end MVS framework. Source code: https://github.com/oppo-us-research/PlaneMVS.

cs.CV↗

arXiv2024

Robust Semi-supervised Learning by Wisely Leveraging Open-set Data

Open-set Semi-supervised Learning (OSSL) holds a realistic setting that unlabeled data may come from classes unseen in the labeled set, i.e., out-of-distribution (OOD) data, which could cause performance degradation in conventional SSL models. To handle this issue, except for the traditional in-distribution (ID) classifier, some existing OSSL approaches employ an extra OOD detection module to avoid the potential negative impact of the OOD data. Nevertheless, these approaches typically employ the entire set of open-set data during their training process, which may contain data unfriendly to the OSSL task that can negatively influence the model performance. This inspires us to develop a robust open-set data selection strategy for OSSL. Through a theoretical understanding from the perspective of learning theory, we propose Wise Open-set Semi-supervised Learning (WiseOpen), a generic OSSL framework that selectively leverages the open-set data for training the model. By applying a gradient-variance-based selection mechanism, WiseOpen exploits a friendly subset instead of the whole open-set dataset to enhance the model's capability of ID classification. Moreover, to reduce the computational expense, we also propose two practical variants of WiseOpen by adopting low-frequency update and loss-based selection respectively. Extensive experiments demonstrate the effectiveness of WiseOpen in comparison with the state-of-the-art.

cs.LG↗

arXiv2024

Modeling and simulation of a mechanism for suppressing the flipping problem of a jumping robot

In order to solve the problem of stable jumping of micro robot, we design a special mechanism: elastic passive joint (EPJ). EPJ can assist in achieving smooth jumping through the opening-closing process when the robot jumps. First, we introduce the composition and operation principle of EPJ, and perform a dynamic modeling of the robot's jumping process. Then, in order to verify the effectiveness of EPJ in controlling the robot's smooth jump, we design a simulation experiment based on MATLAB. Through comparative experiments, it was proved that EPJ can greatly adjust the angular velocity of the robot and increase the jump distance of the robot. Finally, we analyze each parameter in EPJ and performs parameter optimization. After optimization, EPJ achieves a completely flip-free jump of the robot, laying an important foundation for improving the mobility of micro-robot.

cs.RO↗