arXiv subjects

Jaeyeon Kim

Publications and source records attributed to Jaeyeon Kim.

At least 55 records · Page 3Linked to original sources

arXiv2024

Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning

In this technical report, we describe our submission to DCASE2024 Challenge Task6 (Automated Audio Captioning) and Task8 (Language-based Audio Retrieval). We develop our approach building upon the EnCLAP audio captioning framework and optimizing it for Task6 of the challenge. Notably, we outline the changes in the underlying components and the incorporation of the reranking process. Additionally, we submit a supplementary retriever model, a byproduct of our modified framework, to Task8. Our proposed systems achieve FENSE score of 0.542 on Task6 and mAP@10 score of 0.386 on Task8, significantly outperforming the baseline models.

eess.AS↗

arXiv2024

EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance

In this work, we aim to analyze and optimize the EnCLAP framework, a state-of-the-art model in automated audio captioning. We investigate the impact of modifying the acoustic encoder components, explore pretraining with different dataset scales, and study the effectiveness of a reranking scheme. Through extensive experimentation and quantitative analysis of generated captions, we develop EnCLAP++, an enhanced version that significantly surpasses the original.

eess.AS↗

arXiv2024

Mirror Duality in Convex Optimization

While first-order optimization methods are usually designed to efficiently reduce the function value $f(x)$, there has been recent interest in methods efficiently reducing the magnitude of $\nabla f(x)$, and the findings show that the two types of methods exhibit a certain symmetry. In this work, we present mirror duality, a one-to-one correspondence between mirror-descent-type methods reducing function value and reducing gradient magnitude. Using mirror duality, we obtain the dual accelerated mirror descent (dual-AMD) method that efficiently reduces $ψ^*(\nabla f(x))$, where $ψ$ is a distance-generating function and $ψ^*$ quantifies the magnitude of $\nabla f(x)$. We then apply dual-AMD to efficiently reduce $\|\nabla f(\cdot) \|_q$ for $q\in [2,\infty)$ and to efficiently compute $\varepsilon$-approximate solutions of the optimal transport problem.

math.OC↗

arXiv2024

PHANGS-JWST: Data Processing Pipeline and First Full Public Data Release

The exquisite angular resolution and sensitivity of JWST is opening a new window for our understanding of the Universe. In nearby galaxies, JWST observations are revolutionizing our understanding of the first phases of star formation and the dusty interstellar medium. Nineteen local galaxies spanning a range of properties and morphologies across the star-forming main sequence have been observed as part of the PHANGS-JWST Cycle 1 Treasury program at spatial scales of $\sim$5-50pc. Here, we describe pjpipe, an image processing pipeline developed for the PHANGS-JWST program that wraps around and extends the official JWST pipeline. We release this pipeline to the community as it contains a number of tools generally useful for JWST NIRCam and MIRI observations. Particularly for extended sources, pjpipe products provide significant improvements over mosaics from the MAST archive in terms of removing instrumental noise in NIRCam data, background flux matching, and calibration of relative and absolute astrometry. We show that slightly smoothing F2100W MIRI data to 0.9" (degrading the resolution by about 30 percent) reduces the noise by a factor of $\approx$3. We also present the first public release (DR1.1.0) of the pjpipe processed eight-band 2-21 $μ$m imaging for all nineteen galaxies in the PHANGS-JWST Cycle 1 Treasury program. An additional 55 galaxies will soon follow from a new PHANGS-JWST Cycle 2 Treasury program.

astro-ph.GA↗

arXiv2024

Optimal Acceleration for Minimax and Fixed-Point Problems is Not Unique

Recently, accelerated algorithms using the anchoring mechanism for minimax optimization and fixed-point problems have been proposed, and matching complexity lower bounds establish their optimality. In this work, we present the surprising observation that the optimal acceleration mechanism in minimax optimization and fixed-point problems is not unique. Our new algorithms achieve exactly the same worst-case convergence rates as existing anchor-based methods while using materially different acceleration mechanisms. Specifically, these new algorithms are dual to the prior anchor-based accelerated methods in the sense of H-duality. This finding opens a new avenue of research on accelerated algorithms since we now have a family of methods that empirically exhibit varied characteristics while having the same optimal worst-case guarantee.

math.OC↗

arXiv2024

Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates

Neural radiance field is an emerging rendering method that generates high-quality multi-view consistent images from a neural scene representation and volume rendering. Although neural radiance field-based techniques are robust for scene reconstruction, their ability to add or remove objects remains limited. This paper proposes a new language-driven approach for object manipulation with neural radiance fields through dataset updates. Specifically, to insert a new foreground object represented by a set of multi-view images into a background radiance field, we use a text-to-image diffusion model to learn and generate combined images that fuse the object of interest into the given background across views. These combined images are then used for refining the background radiance field so that we can render view-consistent images containing both the object and the background. To ensure view consistency, we propose a dataset updates strategy that prioritizes radiance field training with camera views close to the already-trained views prior to propagating the training to remaining views. We show that under the same dataset updates strategy, we can easily adapt our method for object insertion using data from text-to-3D models as well as object removal. Experimental results show that our method generates photorealistic images of the edited scenes, and outperforms state-of-the-art methods in 3D reconstruction and neural radiance field blending.

cs.CV↗

arXiv2024

Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations

We propose a framework to learn semantics from raw audio signals using two types of representations, encoding contextual and phonetic information respectively. Specifically, we introduce a speech-to-unit processing pipeline that captures two types of representations with different time resolutions. For the language model, we adopt a dual-channel architecture to incorporate both types of representation. We also present new training objectives, masked context reconstruction and masked context prediction, that push models to learn semantics effectively. Experiments on the sSIMI metric of Zero Resource Speech Benchmark 2021 and Fluent Speech Command dataset show our framework learns semantics better than models trained with only one type of representation.

eess.AS↗

arXiv2024

EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning

We propose EnCLAP, a novel framework for automated audio captioning. EnCLAP employs two acoustic representation models, EnCodec and CLAP, along with a pretrained language model, BART. We also introduce a new training objective called masked codec modeling that improves acoustic awareness of the pretrained language model. Experimental results on AudioCaps and Clotho demonstrate that our model surpasses the performance of baseline models. Source code will be available at https://github.com/jaeyeonkim99/EnCLAP . An online demo is available at https://huggingface.co/spaces/enclap-team/enclap .

eess.AS↗

arXiv2023

Time-Reversed Dissipation Induces Duality Between Minimizing Gradient Norm and Function Value

In convex optimization, first-order optimization methods efficiently minimizing function values have been a central subject study since Nesterov's seminal work of 1983. Recently, however, Kim and Fessler's OGM-G and Lee et al.'s FISTA-G have been presented as alternatives that efficiently minimize the gradient magnitude instead. In this paper, we present H-duality, which represents a surprising one-to-one correspondence between methods efficiently minimizing function values and methods efficiently minimizing gradient magnitude. In continuous-time formulations, H-duality corresponds to reversing the time dependence of the dissipation/friction term. To the best of our knowledge, H-duality is different from Lagrange/Fenchel duality and is distinct from any previously known duality or symmetry relations. Using H-duality, we obtain a clearer understanding of the symmetry between Nesterov's method and OGM-G, derive a new class of methods efficiently reducing gradient magnitudes of smooth convex functions, and find a new composite minimization method that is simpler and faster than FISTA-G.

math.OC↗

arXiv2023

PITS: Variational Pitch Inference without Fundamental Frequency for End-to-End Pitch-controllable TTS

Previous pitch-controllable text-to-speech (TTS) models rely on directly modeling fundamental frequency, leading to low variance in synthesized speech. To address this issue, we propose PITS, an end-to-end pitch-controllable TTS model that utilizes variational inference to model pitch. Based on VITS, PITS incorporates the Yingram encoder, the Yingram decoder, and adversarial training of pitch-shifted synthesis to achieve pitch-controllability. Experiments demonstrate that PITS generates high-quality speech that is indistinguishable from ground truth speech and has high pitch-controllability without quality degradation. Code, audio samples, and demo are available at https://github.com/anonymous-pits/pits.

eess.AS↗

arXiv2023

PHANGS-JWST First Results: Spurring on Star Formation: JWST Reveals Localised Star Formation in a Spiral Arm Spur of NGC 628

We combine JWST observations with ALMA CO and VLT-MUSE H$α$ data to examine off-spiral arm star formation in the face-on, grand-design spiral galaxy NGC 628. We focus on the northern spiral arm, around a galactocentric radius of 3-4 kpc, and study two spurs. These form an interesting contrast, as one is CO-rich and one CO-poor, and they have a maximum azimuthal offset in MIRI 21$μ$m and MUSE H$α$ of around 40$^\circ$ (CO-rich) and 55$^\circ$ (CO-poor) from the spiral arm. The star formation rate is higher in the regions of the spurs near to spiral arms, but the star formation efficiency appears relatively constant. Given the spiral pattern speed and rotation curve of this galaxy and assuming material exiting the arms undergoes purely circular motion, these offsets would be reached in 100-150 Myr, significantly longer than the 21$μ$m and H$α$ star formation timescales (both <10 Myr). The invariance of the star formation efficiency in the spurs versus the spiral arms indicates massive star formation is not only triggered in spiral arms, and cannot simply occur in the arms and then drift away from the wave pattern. These early JWST results show that in-situ star formation likely occurs in the spurs, and that the observed young stars are not simply the `leftovers' of stellar birth in the spiral arms. The excellent physical resolution and sensitivity that JWST can attain in nearby galaxies will well resolve individual star-forming regions and help us to better understand the earliest phases of star formation.

astro-ph.GA↗

arXiv2023

PHANGS-JWST First Results: Mid-infrared emission traces both gas column density and heating at 100 pc scales

We compare mid-infrared (mid-IR), extinction-corrected H$α$, and CO (2-1) emission at 70--160 pc resolution in the first four PHANGS-JWST targets. We report correlation strengths, intensity ratios, and power law fits relating emission in JWST's F770W, F1000W, F1130W, and F2100W bands to CO and H$α$. At these scales, CO and H$α$ each correlate strongly with mid-IR emission, and these correlations are each stronger than the one relating CO to H$α$ emission. This reflects that mid-IR emission simultaneously acts as a dust column density tracer, leading to the good match with the molecular gas-tracing CO, and as a heating tracer, leading to the good match with the H$α$. By combining mid-IR, CO, and H$α$ at scales where the overall correlation between cold gas and star formation begins to break down, we are able to separate these two effects. We model the mid-IR above $I_ν= 0.5$~MJy sr$^{-1}$ at F770W, a cut designed to select regions where the molecular gas dominates the interstellar medium (ISM) mass. This bright emission can be described to first order by a model that combines a CO-tracing component and an H$α$-tracing component. The best-fitting models imply that $\sim 50\%$ of the mid-IR flux arises from molecular gas heated by the diffuse interstellar radiation field, with the remaining $\sim 50\%$ associated with bright, dusty star forming regions. We discuss differences between the F770W, F1000W, F1130W bands and the continuum dominated F2100W band and suggest next steps for using the mid-IR as an ISM tracer.

astro-ph.GA↗

arXiv2022

PHANGS-JWST First Results: A Global and Moderately Resolved View of Mid-Infrared and CO Line Emission from Galaxies at the Start of the JWST Era

We explore the relationship between mid-infrared (mid-IR) and CO rotational line emission from massive star-forming galaxies, which is one of the tightest scalings in the local universe. We assemble a large set of unresolved and moderately ($\sim 1$ kpc) spatially resolved measurements of CO (1-0) and CO (2-1) intensity, $I_{\rm CO}$, and mid-IR intensity, $I_{\rm MIR}$, at 8, 12, 22, and 24$μ$m. The $I_{\rm CO}$ vs. $I_{\rm MIR}$ relationship is reasonably described by a power law with slopes $0.7{-}1.2$ and normalization $I_{\rm CO} \sim 1$ K km s$^{-1}$ at $I_{\rm MIR} \sim 1$ MJy sr$^{-1}$. Both the slopes and intercepts vary systematically with choice of line and band. The comparison between the relations measured for CO~(1-0) and CO (2-1) allow us to infer that $R_{21} \propto I_{\rm MIR}^{0.2}$, in good agreement with other work. The $8μ$m and $12μ$m bands, with strong PAH features, show steeper CO vs. mid-IR slopes than the $22μ$m and $24μ$m, consistent with PAH emission arising not just from CO-bright gas but also from atomic or CO-dark gas. The CO-to-mid-IR ratio correlates with global galaxy stellar mass ($M_\star$) and anti-correlates with SFR/$M_\star$. At $\sim 1$ kpc resolution, the first four PHANGS-JWST targets show CO to mid-IR relationships that are quantitatively similar to our larger literature sample, including showing the steep CO-to-mid-IR slopes for the JWST PAH-tracing bands, although we caution that these initial data have a small sample size and span a limited range of intensities.

astro-ph.GA↗

arXiv2022

PHANGS-JWST First Results: Tracing the Diffuse ISM with JWST Imaging of Polycyclic Aromatic Hydrocarbon Emission in Nearby Galaxies

JWST observations of polycyclic aromatic hydrocarbon (PAH) emission provide some of the deepest and highest resolution views of the cold interstellar medium (ISM) in nearby galaxies. If PAHs are well mixed with the atomic and molecular gas and illuminated by the average diffuse interstellar radiation field, PAH emission may provide an approximately linear, high resolution, high sensitivity tracer of diffuse gas surface density. We present a pilot study that explores using PAH emission in this way based on MIRI observations of IC 5332, NGC 628, NGC 1365, and NGC 7496 from the PHANGS-JWST Treasury. Using scaling relationships calibrated in Leroy et al. (2022), scaled F1130W provides 10--40 pc resolution and 3$σ$ sensitivity of $Σ_{\rm gas} \sim 2$ M$_\odot$ pc$^{-2}$. We characterize the surface densities of structures seen at $< 7$ M$_\odot$ pc$^{-2}$ in our targets, where we expect the gas to be HI-dominated. We highlight the existence of filaments, inter-arm emission, and holes in the diffuse ISM at these low surface densities. Below $\sim 10$ M$_\odot$ pc$^{-2}$ for NGC 628, NGC 1365, and NGC 7496 the gas distribution shows a ``Swiss cheese''-like topology due to holes and bubbles pervading the relatively smooth distribution of diffuse ISM. Comparing to recent galaxy simulations, we observe similar topology for the low surface density gas, though with notable variations between simulations with different setups and resolution. Such a comparison of high resolution, low surface density gas with simulations is not possible with existing atomic and molecular gas maps, highlighting the unique power of JWST maps of PAH emission.

astro-ph.GA↗

arXiv2022

PHANGS-JWST First Results: Rapid Evolution of Star Formation in the Central Molecular Gas Ring of NGC1365

Large-scale bars can fuel galaxy centers with molecular gas, often leading to the development of dense ring-like structures where intense star formation occurs, forming a very different environment compared to galactic disks. We pair ~0.3" (30pc) resolution new JWST/MIRI imaging with archival ALMA CO(2-1) mapping of the central ~5kpc of the nearby barred spiral galaxy NGC1365, to investigate the physical mechanisms responsible for this extreme star formation. The molecular gas morphology is resolved into two well-known bright bar lanes that surround a smooth dynamically cold gas disk (R_gal ~ 475pc) reminiscent of non-star-forming disks in early type galaxies and likely fed by gas inflow triggered by stellar feedback in the lanes. The lanes host a large number of JWST-identified massive young star clusters. We find some evidence for temporal star formation evolution along the ring. The complex kinematics in the gas lanes reveal strong streaming motions and may be consistent with convergence of gas streamlines expected there. Indeed, the extreme line-widths are found to be the result of inter-`cloud' motion between gas peaks; ScousePy decomposition reveals multiple components with line widths of ~ 19km/s and surface densities of ~ 800M_sun/pc^2, similar to the properties observed throughout the rest of the central molecular gas structure. Tailored hydro-dynamical simulations exhibit many of the observed properties and imply that the observed structures are transient and highly time-variable. From our study of NGC1365, we conclude that it is predominantly the high gas inflow triggered by the bar that is setting the star formation in its CMZ.

astro-ph.GA↗

arXiv2022

PHANGS-JWST First Results: Duration of the early phase of massive star formation in NGC628

The earliest stages of star formation, when young stars are still deeply embedded in their natal clouds, represent a critical phase in the matter cycle between gas clouds and young stellar regions. Until now, the high-resolution infrared observations required for characterizing this heavily obscured phase (during which massive stars have formed, but optical emission is not detected) could only be obtained for a handful of the most nearby galaxies. One of the main hurdles has been the limited angular resolution of the Spitzer Space Telescope. With the revolutionary capabilities of the JWST, it is now possible to investigate the matter cycle during the earliest phases of star formation as a function of the galactic environment. In this Letter, we demonstrate this by measuring the duration of the embedded phase of star formation and the implied time over which molecular clouds remain inert in the galaxy NGC628 at a distance of 9.8Mpc, demonstrating that the cosmic volume where this measurement can be made has increased by a factor of $>100$ compared to Spitzer. We show that young massive stars remain embedded for $5.1_{-1.4}^{+2.7}$Myr ($2.3_{-1.4}^{+2.7}$Myr of which being heavily obscured), representing $\sim20\%$ of the total cloud lifetime. These values are in broad agreement with previous measurements in five nearby ($D < 3.5$Mpc) galaxies and constitute a proof of concept for the systematic characterization of the early phase of star formation across the nearby galaxy population with the PHANGS-JWST survey.

astro-ph.GA↗

arXiv2022

The PHANGS-JWST Treasury Survey: Star Formation, Feedback, and Dust Physics at High Angular resolution in Nearby GalaxieS

The PHANGS collaboration has been building a reference dataset for the multi-scale, multi-phase study of star formation and the interstellar medium in nearby galaxies. With the successful launch and commissioning of JWST, we can now obtain high-resolution infrared imaging to probe the youngest stellar populations and dust emission on the scales of star clusters and molecular clouds ($\sim$5-50 pc). In Cycle 1, PHANGS is conducting an 8-band imaging survey from 2-21$μ$m of 19 nearby spiral galaxies. CO(2-1) mapping, optical integral field spectroscopy, and UV-optical imaging for all 19 galaxies have been obtained through large programs with ALMA, VLT/MUSE, and Hubble. PHANGS-JWST enables a full inventory of star formation, accurate measurement of the mass and age of star clusters, identification of the youngest embedded stellar populations, and characterization of the physical state of small dust grains. When combined with Hubble catalogs of $\sim$10,000 star clusters, MUSE spectroscopic mapping of $\sim$20,000 HII regions, and $\sim$12,000 ALMA-identified molecular clouds, it becomes possible to measure the timescales and efficiencies of the earliest phases of star formation and feedback, build an empirical model of the dependence of small dust grain properties on local ISM conditions, and test our understanding of how dust-reprocessed starlight traces star formation activity, all across a diversity of galactic environments. Here we describe the PHANGS-JWST Treasury survey, present the remarkable imaging obtained in the first few months of science operations, and provide context for the initial results presented in the first series of PHANGS-JWST publications.

astro-ph.GA↗

arXiv2022

PHANGS-JWST First Results: The 21 $μ$m Compact Source Population

We use PHANGS-JWST data to identify and classify 1271 compact 21 $μ$m sources in four nearby galaxies using MIRI F2100W data. We identify sources using a dendrogram-based algorithm, and we measure the background-subtracted flux densities for JWST bands from 2 $μ$m to 21 $μ$m. Using the SED in JWST as well as HST bands, plus ALMA and MUSE/VLT observations, we classify the sources by eye. Then we use this classification to define regions in color-color space, and so establish a quantitative framework for classifying sources. We identify 1085 sources as belonging to the ISM of the target galaxies with the remainder being dusty stars or background galaxies. These 21 $μ$m sources are strongly spatially associated with HII regions ($>92\%$ of sources), while 74$\%$ of sources are coincident with a stellar association defined in the HST data. Using SED fitting, we find that the stellar masses of the 21 $μ$m sources span a range of 10$^{2}$ to 10$^{4}~M_\odot$ with mass-weighted ages down to 2 Myr. There is a tight correlation between attenuation-corrected H$α$ and 21 $μ$m luminosity for $L_{ν,\mathrm{F2100W}}>10^{19}~\mathrm{W~Hz}^{-1}$. Young embedded source candidates selected at 21 $μ$m are found below this threshold and have $M_\star < 10^{3}~M_\odot$.

astro-ph.GA↗