arXiv subjects

Yi Ren

Publications and source records attributed to Yi Ren.

At least 109 records · Page 6Linked to original sources

arXiv2024

Generative Expressive Conversational Speech Synthesis

Conversational Speech Synthesis (CSS) aims to express a target utterance with the proper speaking style in a user-agent conversation setting. Existing CSS methods employ effective multi-modal context modeling techniques to achieve empathy understanding and expression. However, they often need to design complex network architectures and meticulously optimize the modules within them. In addition, due to the limitations of small-scale datasets containing scripted recording styles, they often fail to simulate real natural conversational styles. To address the above issues, we propose a novel generative expressive CSS system, termed GPT-Talker.We transform the multimodal information of the multi-turn dialogue history into discrete token sequences and seamlessly integrate them to form a comprehensive user-agent dialogue context. Leveraging the power of GPT, we predict the token sequence, that includes both semantic and style knowledge, of response for the agent. After that, the expressive conversational speech is synthesized by the conversation-enriched VITS to deliver feedback to the user.Furthermore, we propose a large-scale Natural CSS Dataset called NCSSD, that includes both naturally recorded conversational speech in improvised styles and dialogues extracted from TV shows. It encompasses both Chinese and English languages, with a total duration of 236 hours.We conducted comprehensive experiments on the reliability of the NCSSD and the effectiveness of our GPT-Talker. Both subjective and objective evaluations demonstrate that our model outperforms other state-of-the-art CSS systems significantly in terms of naturalness and expressiveness. The Code, Dataset, and Pre-trained Model are available at: https://github.com/AI-S2-Lab/GPT-Talker.

cs.CL↗

arXiv2024

Data-driven stellar intrinsic colors and dust reddenings for spectro-photometric data: From the blue-edge method to a machine-learning approach

Intrinsic colors (ICs) of stars are essential for the studies on both stellar physics and dust reddening. In this work, we developed an XGBoost model to predict the ICs with the atmospheric parameters $T_{\rm eff}$, ${\rm log}\,g$, and $\rm [M/H]$. The model was trained and tested for three colors at Gaia and 2MASS bands with 1,040,446 low-reddening sources. The atmospheric parameters were determined by the Gaia DR3 GSP-phot module and were validated by comparing with APOGEE and LAMOST. We further confirmed that the biases in GSP-phot parameters, especially for $\rm [M/H]$, do not present a significant impact on the IC prediction. The generalization error of the model estimated by the test set is 0.014 mag for $(G_{\rm BP}\,{-}\,G_{\rm RP})_0$, 0.050 mag for $(G_{\rm BP}\,{-}\,K_{\rm S})_0$, and 0.040 mag for $(J\,{-}\,K_{\rm S})_0$. The model was applied to a sample containing 5,714,528 reddened stars with stellar parameters from Andrae et al. (2023) to calculate ICs and reddenings. The high consistency in the comparison of $E(J\,{-}\,K_{\rm S})$ between our results and literature values further validates the accuracy of the XGBoost model. The variation of $E(G_{\rm BP}\,{-}\,K_{\rm S})/E(G_{\rm BP}\,{-}\,G_{\rm RP})$, a representation of the extinction law, with Galactic longitude is found on large scales. This work preliminarily presents the feasibility and the accuracy of the machine-learning approach for IC and dust reddening calculation, whose products could be widely applied to spectro-photometric data. The data sets and trained model can be accessed via \url{https://doi.org/10.5281/zenodo.12787594}. The models for more bands will be completed in the following works.

astro-ph.SR↗

arXiv2024

RefXVC: Cross-Lingual Voice Conversion with Enhanced Reference Leveraging

This paper proposes RefXVC, a method for cross-lingual voice conversion (XVC) that leverages reference information to improve conversion performance. Previous XVC works generally take an average speaker embedding to condition the speaker identity, which does not account for the changing timbre of speech that occurs with different pronunciations. To address this, our method uses both global and local speaker embeddings to capture the timbre changes during speech conversion. Additionally, we observed a connection between timbre and pronunciation in different languages and utilized this by incorporating a timbre encoder and a pronunciation matching network into our model. Furthermore, we found that the variation in tones is not adequately reflected in a sentence, and therefore, we used multiple references to better capture the range of a speaker's voice. The proposed method outperformed existing systems in terms of both speech quality and speaker similarity, highlighting the effectiveness of leveraging reference information in cross-lingual voice conversion. The converted speech samples can be found on the website: \url{http://refxvc.dn3point.com}

eess.AS↗

arXiv2024

State-Constrained Zero-Sum Differential Games with One-Sided Information

We study zero-sum differential games with state constraints and one-sided information, where the informed player (Player 1) has a categorical payoff type unknown to the uninformed player (Player 2). The goal of Player 1 is to minimize his payoff without violating the constraints, while that of Player 2 is to violate the state constraints if possible, or to maximize the payoff otherwise. One example of the game is a man-to-man matchup in football. Without state constraints, Cardaliaguet (2007) showed that the value of such a game exists and is convex to the common belief of players. Our theoretical contribution is an extension of this result to games with state constraints and the derivation of the primal and dual subdynamic principles necessary for computing behavioral strategies. Different from existing works that are concerned about the scalability of no-regret learning in games with discrete dynamics, our study reveals the underlying structure of strategies for belief manipulation resulting from information asymmetry and state constraints. This structure will be necessary for scalable learning on games with continuous actions and long time windows. We use a simplified football game to demonstrate the utility of this work, where we reveal player positions and belief states in which the attacker should (or should not) play specific random deceptive moves to take advantage of information asymmetry, and compute how the defender should respond.

cs.GT↗

arXiv2024

Pontryagin Neural Operator for Solving Parametric General-Sum Differential Games

The values of two-player general-sum differential games are viscosity solutions to Hamilton-Jacobi-Isaacs (HJI) equations. Value and policy approximations for such games suffer from the curse of dimensionality (CoD). Alleviating CoD through physics-informed neural networks (PINN) encounters convergence issues when differentiable values with large Lipschitz constants are present due to state constraints. On top of these challenges, it is often necessary to learn generalizable values and policies across a parametric space of games, e.g., for game parameter inference when information is incomplete. To address these challenges, we propose in this paper a Pontryagin-mode neural operator that outperforms the current state-of-the-art hybrid PINN model on safety performance across games with parametric state constraints. Our key contribution is the introduction of a costate loss defined on the discrepancy between forward and backward costate rollouts, which are computationally cheap. We show that the costate dynamics, which can reflect state constraint violation, effectively enables the learning of differentiable values with large Lipschitz constants, without requiring manually supervised data as suggested by the hybrid PINN model. More importantly, we show that the close relationship between costates and policies makes the former critical in learning feedback control policies with generalizable safety performance.

cs.LG↗

arXiv2024

The RoboDrive Challenge: Drive Anytime Anywhere in Any Condition

In the realm of autonomous driving, robust perception under out-of-distribution conditions is paramount for the safe deployment of vehicles. Challenges such as adverse weather, sensor malfunctions, and environmental unpredictability can severely impact the performance of autonomous systems. The 2024 RoboDrive Challenge was crafted to propel the development of driving perception technologies that can withstand and adapt to these real-world variabilities. Focusing on four pivotal tasks -- BEV detection, map segmentation, semantic occupancy prediction, and multi-view depth estimation -- the competition laid down a gauntlet to innovate and enhance system resilience against typical and atypical disturbances. This year's challenge consisted of five distinct tracks and attracted 140 registered teams from 93 institutes across 11 countries, resulting in nearly one thousand submissions evaluated through our servers. The competition culminated in 15 top-performing solutions, which introduced a range of innovative approaches including advanced data augmentation, multi-sensor fusion, self-supervised learning for error correction, and new algorithmic strategies to enhance sensor robustness. These contributions significantly advanced the state of the art, particularly in handling sensor inconsistencies and environmental variability. Participants, through collaborative efforts, pushed the boundaries of current technologies, showcasing their potential in real-world scenarios. Extensive evaluations and analyses provided insights into the effectiveness of these solutions, highlighting key trends and successful strategies for improving the resilience of driving perception systems. This challenge has set a new benchmark in the field, providing a rich repository of techniques expected to guide future research in this field.

cs.CV↗

arXiv2024

Hacking Task Confounder in Meta-Learning

Meta-learning enables rapid generalization to new tasks by learning knowledge from various tasks. It is intuitively assumed that as the training progresses, a model will acquire richer knowledge, leading to better generalization performance. However, our experiments reveal an unexpected result: there is negative knowledge transfer between tasks, affecting generalization performance. To explain this phenomenon, we conduct Structural Causal Models (SCMs) for causal analysis. Our investigation uncovers the presence of spurious correlations between task-specific causal factors and labels in meta-learning. Furthermore, the confounding factors differ across different batches. We refer to these confounding factors as "Task Confounders". Based on these findings, we propose a plug-and-play Meta-learning Causal Representation Learner (MetaCRL) to eliminate task confounders. It encodes decoupled generating factors from multiple tasks and utilizes an invariant-based bi-level optimization mechanism to ensure their causality for meta-learning. Extensive experiments on various benchmark datasets demonstrate that our work achieves state-of-the-art (SOTA) performance.

cs.LG↗

arXiv2024

lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning

Although much research has been done on proposing new models or loss functions to improve the generalisation of artificial neural networks (ANNs), less attention has been directed to the impact of the training data on generalisation. In this work, we start from approximating the interaction between samples, i.e. how learning one sample would modify the model's prediction on other samples. Through analysing the terms involved in weight updates in supervised learning, we find that labels influence the interaction between samples. Therefore, we propose the labelled pseudo Neural Tangent Kernel (lpNTK) which takes label information into consideration when measuring the interactions between samples. We first prove that lpNTK asymptotically converges to the empirical neural tangent kernel in terms of the Frobenius norm under certain assumptions. Secondly, we illustrate how lpNTK helps to understand learning phenomena identified in previous work, specifically the learning difficulty of samples and forgetting events during learning. Moreover, we also show that using lpNTK to identify and remove poisoning training samples does not hurt the generalisation performance of ANNs.

cs.LG↗

arXiv2024

Value Approximation for Two-Player General-Sum Differential Games with State Constraints

Solving Hamilton-Jacobi-Isaacs (HJI) PDEs numerically enables equilibrial feedback control in two-player differential games, yet faces the curse of dimensionality (CoD). While physics-informed neural networks (PINNs) have shown promise in alleviating CoD in solving PDEs, vanilla PINNs fall short in learning discontinuous solutions due to their sampling nature, leading to poor safety performance of the resulting policies when values are discontinuous due to state or temporal logic constraints. In this study, we explore three potential solutions to this challenge: (1) a hybrid learning method that is guided by both supervisory equilibria and the HJI PDE, (2) a value-hardening method where a sequence of HJIs are solved with increasing Lipschitz constant on the constraint violation penalty, and (3) the epigraphical technique that lifts the value to a higher dimensional state space where it becomes continuous. Evaluations through 5D and 9D vehicle and 13D drone simulations reveal that the hybrid method outperforms others in terms of generalization and safety performance by taking advantage of both the supervisory equilibrium values and costates, and the low cost of PINN loss gradients.

cs.RO↗

arXiv2024

Modeling of Granulation in Red Supergiants in the Magellanic Clouds with the Gaussian Process Regressions

The granulation of red supergiants (RSGs) in the Magellanic Clouds are systematically investigated by combining the latest RSGs samples and light curves from the Optical Gravitational Lensing Experiment and the All-Sky Automated Survey for Supernovae. The present RSGs samples are firstly examined for foreground stars and possible misidentified sources, and the light curves are sequentially checked to remove the outliers by white noise and photometric quality. The Gaussian Process regression is used to model the granulation, and the Markov Chain Monte Carlo is applied to derive the granulation amplitude $σ$ and the period of the undamped oscillator $ρ$, as well as the damping timescale $τ$. The dimensionless quality factor $Q$ is then calculated through $Q=πτ/ρ$. RSGs around $Q = 1/\sqrt{2}$ are considered to have significant granulation signals and are used for further analysis. Combining granulation parameters with stellar parameters, robust scaling relations for the timescale $ρ$ are established, while the scaling relations for amplitude $σ$ are represented by a piecewise function, possibly related to the tendency of amplitudes in faint RSGs to converge towards a certain value. Comparing results between the SMC and LMC confirms that amplitudes and timescales become larger with metallicity. In examining the scaling relations between the two galaxies, it is found that $ρ$ is nearly independent of metallicity, whereas $σ$ is more significantly affected by metallicity. The Gaussian Process method is compared with the periodogram fitting of the granulations, and the advantages of either are discussed.

astro-ph.SR↗

arXiv2024

Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis

Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech prompts, which significantly reduces the data and computation requirements for voice cloning by skipping the fine-tuning process. However, the prompting mechanisms of zero-shot TTS still face challenges in the following aspects: 1) previous works of zero-shot TTS are typically trained with single-sentence prompts, which significantly restricts their performance when the data is relatively sufficient during the inference stage. 2) The prosodic information in prompts is highly coupled with timbre, making it untransferable to each other. This paper introduces Mega-TTS 2, a generic prompting mechanism for zero-shot TTS, to tackle the aforementioned challenges. Specifically, we design a powerful acoustic autoencoder that separately encodes the prosody and timbre information into the compressed latent space while providing high-quality reconstructions. Then, we propose a multi-reference timbre encoder and a prosody latent language model (P-LLM) to extract useful information from multi-sentence prompts. We further leverage the probabilities derived from multiple P-LLM outputs to produce transferable and controllable prosody. Experimental results demonstrate that Mega-TTS 2 could not only synthesize identity-preserving speech with a short prompt of an unseen speaker from arbitrary sources but consistently outperform the fine-tuning method when the volume of data ranges from 10 seconds to 5 minutes. Furthermore, our method enables to transfer various speaking styles to the target timbre in a fine-grained and controlled manner. Audio samples can be found in https://boostprompt.github.io/boostprompt/.

eess.AS↗

arXiv2024

Searching for short-period variables in M31: method and catalogs

Utilizing high-cadence and continuous g- and r-band data over three nights acquired from the 3.6-meter Canada France Hawaii Telescope (CFHT) aimed to find short-duration microlensing events, we conduct a systematic search for variables, transients, and asteroids across a $\sim1^\circ$ field of view of the Andromeda Galaxy (M 31). We present a catalog of 5859 variable stars, yielding the most extensive compilation of short-period variable sources of M 31. We also detected 19 flares, predominantly associated with foreground M dwarfs in the Milky Way. In addition, we discovered 17 previously unknown asteroid candidates, and we subsequently reported them to the Minor Planet Center. Lastly, we report a microlensing event candidate C-ML-1 and present a preliminary analysis.

astro-ph.SR↗

arXiv2024

ModaLink: Unifying Modalities for Efficient Image-to-PointCloud Place Recognition

Place recognition is an important task for robots and autonomous cars to localize themselves and close loops in pre-built maps. While single-modal sensor-based methods have shown satisfactory performance, cross-modal place recognition that retrieving images from a point-cloud database remains a challenging problem. Current cross-modal methods transform images into 3D points using depth estimation for modality conversion, which are usually computationally intensive and need expensive labeled data for depth supervision. In this work, we introduce a fast and lightweight framework to encode images and point clouds into place-distinctive descriptors. We propose an effective Field of View (FoV) transformation module to convert point clouds into an analogous modality as images. This module eliminates the necessity for depth estimation and helps subsequent modules achieve real-time performance. We further design a non-negative factorization-based encoder to extract mutually consistent semantic features between point clouds and images. This encoder yields more distinctive global descriptors for retrieval. Experimental results on the KITTI dataset show that our proposed methods achieve state-of-the-art performance while running in real time. Additional evaluation on the HAOMO dataset covering a 17 km trajectory further shows the practical generalization capabilities. We have released the implementation of our methods as open source at: https://github.com/haomo-ai/ModaLink.git.

cs.CV↗

arXiv2024

Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from an unseen image, and then animate it with a reference video or audio to generate a talking portrait video. The existing methods fail to simultaneously achieve the goals of accurate 3D avatar reconstruction and stable talking face animation. Besides, while the existing works mainly focus on synthesizing the head part, it is also vital to generate natural torso and background segments to obtain a realistic talking portrait video. To address these limitations, we present Real3D-Potrait, a framework that (1) improves the one-shot 3D reconstruction power with a large image-to-plane model that distills 3D prior knowledge from a 3D face generative model; (2) facilitates accurate motion-conditioned animation with an efficient motion adapter; (3) synthesizes realistic video with natural torso movement and switchable background using a head-torso-background super-resolution model; and (4) supports one-shot audio-driven talking face generation with a generalizable audio-to-motion model. Extensive experiments show that Real3D-Portrait generalizes well to unseen identities and generates more realistic talking portrait videos compared to previous methods. Video samples and source code are available at https://real3dportrait.github.io .

cs.CV↗

arXiv2024

Resolved Near-infrared Stellar Photometry from the Magellan Telescope for 13 Nearby Galaxies: JAGB Method Distances

We present near-infrared JHK photometry for the resolved stellar populations in 13 nearby galaxies: NGC 6822, IC 1613, NGC 3109, Sextans B, Sextans A, NGC 300, NGC 55, NGC 7793, NGC 247, NGC 5253, Cen A, NGC 1313, and M83, acquired from the 6.5m Baade-Magellan telescope. We measure distances to each galaxy using the J-region asymptotic giant branch (JAGB) method, a new standard candle that leverages the constant luminosities of color-selected, carbon-rich AGB stars. While only single-epoch, random-phase photometry is necessary to derive JAGB distances, our photometry is time-averaged over multiple epochs, thereby decreasing the contribution of the JAGB stars' intrinsic variability to the measured dispersions in their observed luminosity functions. To cross-validate these distances, we also measure near-infrared tip of the red giant branch (TRGB) distances to these galaxies. The residuals obtained from subtracting the distance moduli from the two methods yield an RMS scatter of $σ_{JAGB - TRGB}= \pm 0.07$ mag. Therefore, all systematics in either the JAGB method and TRGB method (e.g., crowding, differential reddening, star formation histories) must be contained within these $\pm0.07$ mag bounds for this sample of galaxies because the JAGB and TRGB distance indicators are drawn from entirely distinct stellar populations, and are thus affected by these systematics independently. Finally, the composite JAGB star luminosity function formed from this diverse sample of galaxies is well-described by a Gaussian function with a modal value of $M_J = -6.20 \pm 0.003$ mag (stat), indicating the underlying JAGB star luminosity function of a well-sampled full star formation history is highly symmetric and Gaussian, based on over 6,700 JAGB stars in the composite sample.

astro-ph.GA↗

arXiv2024

Dance2MIDI: Dance-driven multi-instruments music generation

Dance-driven music generation aims to generate musical pieces conditioned on dance videos. Previous works focus on monophonic or raw audio generation, while the multi-instruments scenario is under-explored. The challenges associated with the dance-driven multi-instrument music (MIDI) generation are twofold: 1) no publicly available multi-instruments MIDI and video paired dataset and 2) the weak correlation between music and video. To tackle these challenges, we build the first multi-instruments MIDI and dance paired dataset (D2MIDI). Based on our proposed dataset, we introduce a multi-instruments MIDI generation framework (Dance2MIDI) conditioned on dance video. Specifically, 1) to capture the relationship between dance and music, we employ the Graph Convolutional Network to encode the dance motion. This allows us to extract features related to dance movement and dance style, 2) to generate a harmonious rhythm, we utilize a Transformer model to decode the drum track sequence, leveraging a cross-attention mechanism, and 3) we model the task of generating the remaining tracks based on the drum track as a sequence understanding and completion task. A BERT-like model is employed to comprehend the context of the entire music piece through self-supervised learning. We evaluate the generated music of our framework trained on the D2MIDI dataset and demonstrate that our method achieves State-of-the-Art performance.

cs.MM↗

arXiv2024

The Star Formation History in Local Group Galaxies. I. Ten Dwarf Galaxies

The star formation histories (SFHs) of galaxies provide valuable insights into galaxy evolution and stellar physics. Understanding the SFHs enables the study of chemical enrichment of galaxies, star formation triggered by interactions, and the behavior of various stellar populations. This work investigates the SFHs of ten dwarf galaxies in the Local Group (LG), which spans a wide range of types, masses, luminosities, and metallicities. The analysis is based on our new sample of the member stars in the LG after removing the foreground dwarf stars by the near-infrared color-color diagram and the Gaia astrometric information. The samples include the most complete and pure red supergiants and asymptotic giant branch stars to gain valuable insights into the recent SFHs of the galaxies. The CMD fitting method is introduced to measure the SFH. The Padova isochrones are used to generate initial model CMDs, accounting for photometric errors and completeness through star field simulations to match the completeness and error distributions of the observed CMDs. Subsequently, the SFHs, distance modulus, and metallicity of the ten dwarf galaxies are determined by fitting the CMDs. The results indicate that the star formation rates (SFRs) of dwarf irregulars show a gradual increase, while those of dwarf ellipticals exhibit a gradual decrease from the past to the present. Furthermore, this work shows that the star formation activity in dwarf ellipticals persisted up to 30 Myr ago. A significant increasing feature in the SFH of NGC 6822 reveals star formation activity triggered by an interaction event.

astro-ph.GA↗

arXiv2024

The Metallicity Distribution in the LMC and the SMC based on the Tip-RGB Colors

The color index $(J-K)_0$ of tip-red giant branch (TRGB) is used to study the metallicity distribution in the Large and Small Magellanic Cloud. With the most complete and pure sample of red member stars so far, the areas are divided into 154 and 70 bins for the LMC and SMC respectively with similar number of stars by the Voronoi binning. For each bin, the position of TRGB on the near-infrared color-magnitude diagram, specifically $(J-K)_0/K_0$, is determined by the Poison-Noise weighted method. Converting the color index of TRGB into metallicity, the metallicity gradients in the LMC and the SMC are obtained in four major directions. For the LMC, the gradient to the north is $-0.006 \pm 0.004$ dex kpc$^{-1}$, to the south $-0.010 \pm 0.005$ dex kpc$^{-1}$, to the east $-0.006 \pm 0.003$ dex kpc$^{-1}$, and to the west $-0.010 \pm 0.003$ dex kpc$^{-1}$. The farthest distance extends to 16 kpc. For the SMC, the gradients to the north, south, east, and west are $-0.017 \pm 0.031$ dex kpc$^{-1}$, $-0.016 \pm 0.007$ dex kpc$^{-1}$, $-0.003 \pm 0.002$ dex kpc$^{-1}$, and $-0.004 \pm 0.003$ dex kpc$^{-1}$, respectively. The farthest distance for the SMC extends to 27 kpc. The gradient is large from the center to 1 kpc.

astro-ph.GA↗