arXiv subjects

Atsuo Hiroe

Publications and source records attributed to Atsuo Hiroe.

6 recordsLinked to original sources

arXiv2026

MMAudio-LABEL: Audio Event Labeling via Audio Generation for Silent Video

Recent advances in multimodal generation have enabled high-quality audio generation from silent videos. Practical applications, such as sound production, demand not only the generated audio but also explicit sound event labels detailing the type and timing of sounds. One straightforward approach involves applying a standard sound event detection to the generated audio. However, this post-hoc pipeline is inherently limited, as it is prone to error accumulation. To address this limitation, we propose MMAudio-LABEL (LAtent-Based Event Labeling), an event-aware audio generation framework built on a foundational audio generation model as its backbone that jointly generates audio and frame-aligned sound event predictions from silent videos. We evaluate our method on the Greatest Hits dataset for onset detection and 17-class material classification. Our approach improves onset-detection accuracy from 46.7% to 75.0% and material-classification accuracy from 40.6% to 61.0% over baselines. These results suggest that jointly learning audio generation and event prediction enables a more interpretable and practical video-to-audio synthesis.

cs.SD↗

arXiv2024

Can all variations within the unified mask-based beamformer framework achieve identical peak extraction performance?

This study investigates mask-based beamformers (BFs), which estimate filters for target sound extraction (TSE) using time-frequency masks. Although multiple mask-based BFs have been proposed, no consensus has been reached on which one offers the best target-extraction performance. Previously, we found that maximum signal-to-noise ratio and minimum mean square error (MSE) BFs can achieve the same extraction performance as the theoretical upper-bound performance, with each BF containing a different optimal mask. However, two issues remained unsolved: only two BFs were covered, excluding the minimum variance distortionless response BF, and ideal scaling (IS) was employed to ideally adjust the output scale, which is not applicable to realistic scenarios. To address these issues, this study proposes a unified framework for mask-based BFs comprising two processes: filter estimation that can cover all possible BFs and scaling applicable to realistic scenarios by employing a mask to generate a scaling reference. Based on the operators and covariance matrices used in BF formulas, all possible BFs can be classified into 12 variations, including two new ones. Optimal masks for both processes are obtained by minimizing the MSE between the target and BF output. The experimental results using the CHiME-4 dataset suggested that 1) all 12 variations can achieve the theoretical upper-bound performance, and 2) mask-based scaling can behave like IS, even when constraining the temporal mean of a non-negative mask to one. These results can be explained by considering the practical parameter count of the masks. These findings contribute to 1) designing a TSE system, 2) improving scaling accuracy through mask-based scaling, and 3) estimating the extraction performance of a BF.

eess.SP↗

arXiv2023

Online Similarity-and-Independence-Aware Beamformer for Low-latency Target Sound Extraction

This study introduces an online target sound extraction (TSE) process using the similarity-and-independence-aware beamformer (SIBF) derived from an iterative batch algorithm. The study aimed to reduce latency while maintaining extraction accuracy. The SIBF, which is a linear method, provides more accurate estimates of the target than an approximate magnitude spectrogram reference. The transition to an online algorithm reduces latency but presents challenges. First, contrary to the conventional assumption, deriving the online algorithm may degrade accuracy as compared to the batch algorithm using a sliding window. Second, conventional post-processing methods intended for scaling the estimated target may widen the accuracy gap between the two algorithms. This study adopts an approach that addresses these challenges and minimizes the accuracy gap during post-processing. It proposes a novel scaling method based on the single-channel Wiener filter (SWF-based scaling). To further improve accuracy, the study introduces a modified version of the time-frequency-varying variance generalized Gaussian distribution as a source model to represent the joint probability between the target and reference. Experimental results using the CHiME-3 dataset demonstrate several key findings: 1) SWF-based scaling effectively eliminates the gap between the two algorithms and improves accuracy. 2) The new source model achieves optimal accuracy, corresponding to the Laplacian model. 3) Our online SIBF outperforms conventional linear TSE methods, including independent vector extraction and minimum mean square error beamforming. These findings can contribute to the fields of beamforming and blind source separation.

eess.AS↗

arXiv2023

Is the Ideal Ratio Mask Really the Best? -- Exploring the Best Extraction Performance and Optimal Mask of Mask-based Beamformers

This study investigates mask-based beamformers (BFs), which estimate filters to extract target speech using time-frequency masks. Although several BF methods have been proposed, the following aspects are yet to be comprehensively investigated. 1) Which BF can provide the best extraction performance in terms of the closeness of the BF output to the target speech? 2) Is the optimal mask for the best performance common for all BFs? 3) Is the ideal ratio mask (IRM) identical to the optimal mask? Accordingly, we investigate these issues considering four mask-based BFs: the maximum signal-to-noise ratio BF, two variants of this, and the multichannel Wiener filter (MWF) BF. To obtain the optimal mask corresponding to the peak performance for each BF, we employ an approach that minimizes the mean square error between the BF output and target speech for each utterance. Via the experiments with the CHiME-3 dataset, we verify that the four BFs have the same peak performance as the upper bound provided by the ideal MWF BF, whereas the optimal mask depends on the adopted BF and differs from the IRM. These observations differ from the conventional idea that the optimal mask is common for all BFs and that peak performance differs for each BF. Hence, this study contributes to the design of mask-based BFs.

eess.AS↗

arXiv2021

Similarity-and-Independence-Aware Beamformer with Iterative Casting and Boost Start for Target Source Extraction Using Reference

Target source extraction is significant for improving human speech intelligibility and the speech recognition performance of computers. This study describes a method for target source extraction, called the similarity-and-independence-aware beamformer (SIBF). The SIBF extracts the target source using a rough magnitude spectrogram as the reference signal. The advantage of the SIBF is that it can obtain a more accurate signal than the spectrogram generated by target-enhancing methods such as speech enhancement based on deep neural networks. For the extraction, we extend the framework of deflationary independent component analysis (ICA) by considering the similarities between the reference and extracted target sources, in addition to the mutual independence of all the potential sources. To solve the extraction problem by maximum-likelihood estimation, we introduce three source models that can reflect the similarities. The major contributions of this study are as follows. First, the extraction performance is improved using two methods, namely boost start for faster convergence and iterative casting for generating a more accurate reference. The effectiveness of these methods is verified through experiments using the CHiME3 dataset. Second, a concept of a fixed point pertaining to accuracy is developed. This concept facilitates understanding the relationship between the reference and SIBF output in terms of accuracy. Third, a unified formulation of the SIBF and mask-based beamformer is realized to apply the expertise of conventional BFs to the SIBF. The findings of this study can also improve the performance of the SIBF and promote research on ICA and conventional beamformers. Index Terms: beamformer, independent component analysis, source separation, speech enhancement, target source extraction

eess.AS↗

arXiv2020

Similarity-and-Independence-Aware Beamformer: Method for Target Source Extraction using Magnitude Spectrogram as Reference

This study presents a novel method for source extraction, referred to as the similarity-and-independence-aware beamformer (SIBF). The SIBF extracts the target signal using a rough magnitude spectrogram as the reference signal. The advantage of the SIBF is that it can obtain an accurate target signal, compared to the spectrogram generated by target-enhancing methods such as the speech enhancement based on deep neural networks (DNNs). For the extraction, we extend the framework of the deflationary independent component analysis, by considering the similarity between the reference and extracted target, as well as the mutual independence of all potential sources. To solve the extraction problem by maximum-likelihood estimation, we introduce two source model types that can reflect the similarity. The experimental results from the CHiME3 dataset show that the target signal extracted by the SIBF is more accurate than the reference signal generated by the DNN. Index Terms: semiblind source separation, similarity-and-independence-aware beamformer, deflationary independent component analysis, source model

eess.AS↗