arXiv subjects

Yusuke Sakai

Publications and source records attributed to Yusuke Sakai.

At least 55 records · Page 3Linked to original sources

arXiv2025

BQA: Body Language Question Answering Dataset for Video Large Language Models

A large part of human communication relies on nonverbal cues such as facial expressions, eye contact, and body language. Unlike language or sign language, such nonverbal communication lacks formal rules, requiring complex reasoning based on commonsense understanding. Enabling current Video Large Language Models (VideoLLMs) to accurately interpret body language is a crucial challenge, as human unconscious actions can easily cause the model to misinterpret their intent. To address this, we propose a dataset, BQA, a body language question answering dataset, to validate whether the model can correctly interpret emotions from short clips of body language comprising 26 emotion labels of videos of body language. We evaluated various VideoLLMs on BQA and revealed that understanding body language is challenging, and our analyses of the wrong answers by VideoLLMs show that certain VideoLLMs made significantly biased answers depending on the age group and ethnicity of the individuals in the video. The dataset is available.

cs.CL↗

arXiv2025

Arcsecond-Scale X-ray Imaging and Spectroscopy of SS 433 with Chandra HETG

We present a spatial and spectral analysis of arcsecond-scale X-ray emission in SS 433 using zeroth-order data from Chandra High-Energy Transmission Grating (HETG) observations. The analysis is based on 24 observations acquired between 1999 and 2024, comprising a total exposure of $\sim$850 ks and covering a wide range of orbital and precessional phases. Among these, the $\sim$140 ks observation from 2014 was analyzed in detail for this study. This data provides the best statistics and was taken when the jets were nearly perpendicular to the line of sight and the accretion disk was eclipsed. By applying an energy-dependent subpixel event repositioning algorithm and the Richardson-Lucy deconvolution, we enhanced the spatial resolution and revealed eastern and western knot-like structures at a distance of $\sim$1.7 arcsec ($\sim 10^{17}$ cm) from the core. These features are consistent with the kinematic precession model, and the positions of the knots suggest that they were ejected approximately 200 days prior to the observation. A comparison with VLA radio data obtained at a similar precessional phase shows that the X-ray emission extends east-west on a scale comparable to that of the radio emission. While the core is bright in both X-rays and radio, the brightness contrast between the knots and the core is smaller in X-rays than in radio. Spatially resolved spectroscopy indicates that prominent Fe lines in the core X-ray spectrum are well explained by thermal plasma emission. In contrast, Fe lines are not evident in the outer regions after accounting for potential core contamination, suggesting a dominant contribution from non-thermal processes. These findings imply that the arcsecond-scale X-ray structures may vary observationally with viewing conditions or precessional phase, but likely reflect a relatively stable jet-driving mechanism operating within the SS 433 system.

astro-ph.HE↗

arXiv2025

Quantum Machine Learning for Identifying Transient Events in X-ray Light Curves

We investigate whether a novel method of quantum machine learning (QML) can identify anomalous events in X-ray light curves as transient events and apply it to detect such events from the XMM-Newton 4XMM-DR14 catalog. The architecture we adopt is a quantum version of the long-short term memory (LSTM) where some fully connected layers are replaced with quantum circuits. The LSTM, making predictions based on preceding data, allows identification of anomalies by comparing predicted and actual time-series data. The necessary training data are generated by simulating active galactic nucleus-like light curves as the species would be a significant population in the XMM-Newton catalog. Additional anomaly data used to assess trained quantum LSTM (QLSTM) models are produced by adding flares like quasi-periodic eruptions to the training data. Comparing various aspects of the performances of the quantum and classical LSTM models, we find that QLSTM models incorporating quantum superposition and entanglement slightly outperform the classical LSTM (CLSTM) model in expressive power, accuracy, and true-positive rate. The highest-performance QLSTM model is then used to identify transient events in 4XMM-DR14. Out of 40154 light curves in the 0.2--12 keV band, we detect 113 light curves with anomalies, or transient event candidates. This number is $\approx$ 1.3 times that of anomalies detectable with the CLSTM model. By utilizing SIMBAD and four wide-field survey catalogs made by ROSAT, SkyMapper, Pan-STARRS, and WISE, no possible counterparts are found for 12 detected anomalies.

astro-ph.HE↗

arXiv2025

Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability

In generative commonsense reasoning tasks such as CommonGen, generative large language models (LLMs) compose sentences that include all given concepts. However, when focusing on instruction-following capabilities, if a prompt specifies a concept order, LLMs must generate sentences that adhere to the specified order. To address this, we propose Ordered CommonGen, a benchmark designed to evaluate the compositional generalization and instruction-following abilities of LLMs. This benchmark measures ordered coverage to assess whether concepts are generated in the specified order, enabling a simultaneous evaluation of both abilities. We conducted a comprehensive analysis using 36 LLMs and found that, while LLMs generally understand the intent of instructions, biases toward specific concept order patterns often lead to low-diversity outputs or identical results even when the concept order is altered. Moreover, even the most instruction-compliant LLM achieved only about 75% ordered coverage, highlighting the need for improvements in both instruction-following and compositional generalization capabilities.

cs.CL↗

arXiv2025

Diversity Explains Inference Scaling Laws: Through a Case Study of Minimum Bayes Risk Decoding

Inference methods play an important role in eliciting the performance of large language models (LLMs). Currently, LLMs use inference methods utilizing generated multiple samples, which can be derived from Minimum Bayes Risk (MBR) Decoding. Previous studies have conducted empirical analyses to clarify the improvements in generation performance achieved by MBR decoding and have reported various observations. However, the theoretical underpinnings of these findings remain uncertain. To address this, we offer a new theoretical interpretation of MBR decoding from the perspective of bias-diversity decomposition. In this interpretation, the error in the quality estimation of hypotheses by MBR decoding is decomposed into two main factors: bias, which considers the closeness between the utility function and human evaluation, and diversity, which represents the variability in the quality estimation of the utility function. The theoretical analysis reveals the difficulty of simultaneously improving bias and diversity, confirming the validity of enhancing MBR decoding performance by increasing diversity. Furthermore, we reveal that diversity can explain one aspect of inference scaling laws that describe performance improvement by increasing sample size. Moreover, experiments across multiple NLP tasks yielded results consistent with these theoretical characteristics. Our code is available at https://github.com/naist-nlp/mbr-bias-diversity.

cs.CL↗

arXiv2025

Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?

One of the goals of automatic evaluation metrics in grammatical error correction (GEC) is to rank GEC systems such that it matches human preferences. However, current automatic evaluations are based on procedures that diverge from human evaluation. Specifically, human evaluation derives rankings by aggregating sentence-level relative evaluation results, e.g., pairwise comparisons, using a rating algorithm, whereas automatic evaluation averages sentence-level absolute scores to obtain corpus-level scores, which are then sorted to determine rankings. In this study, we propose an aggregation method for existing automatic evaluation metrics which aligns with human evaluation methods to bridge this gap. We conducted experiments using various metrics, including edit-based metrics, n-gram based metrics, and sentence-level metrics, and show that resolving the gap improves results for the most of metrics on the SEEDA benchmark. We also found that even BERT-based metrics sometimes outperform the metrics of GPT-4. The proposed ranking method is integrated gec-metrics.

cs.CL↗

arXiv2025

IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator

We propose IMPARA-GED, a novel reference-free automatic grammatical error correction (GEC) evaluation method with grammatical error detection (GED) capabilities. We focus on the quality estimator of IMPARA, an existing automatic GEC evaluation method, and construct that of IMPARA-GED using a pre-trained language model with enhanced GED capabilities. Experimental results on SEEDA, a meta-evaluation dataset for automatic GEC evaluation methods, demonstrate that IMPARA-GED achieves the highest correlation with human sentence-level evaluations.

cs.CL↗

arXiv2025

Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries

Cross-lingual vocabulary transfer plays a promising role in adapting pre-trained language models to new languages, including low-resource languages. Existing approaches that utilize monolingual or parallel corpora face challenges when applied to languages with limited resources. In this work, we propose a simple yet effective vocabulary transfer method that utilizes bilingual dictionaries, which are available for many languages, thanks to descriptive linguists. Our proposed method leverages a property of BPE tokenizers where removing a subword from the vocabulary causes a fallback to shorter subwords. The embeddings of target subwords are estimated iteratively by progressively removing them from the tokenizer. The experimental results show that our approach outperforms existing methods for low-resource languages, demonstrating the effectiveness of a dictionary-based approach for cross-lingual vocabulary transfer.

cs.CL↗

arXiv2025

gec-metrics: A Unified Library for Grammatical Error Correction Evaluation

We introduce gec-metrics, a library for using and developing grammatical error correction (GEC) evaluation metrics through a unified interface. Our library enables fair system comparisons by ensuring that everyone conducts evaluations using a consistent implementation. Moreover, it is designed with a strong focus on API usage, making it highly extensible. It also includes meta-evaluation functionalities and provides analysis and visualization scripts, contributing to developing GEC evaluation metrics. Our code is released under the MIT license and is also distributed as an installable package. The video is available on YouTube.

cs.CL↗

arXiv2025

AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising

With the increase in the fluency of ad texts automatically created by natural language generation technology, there is high demand to verify the quality of these creatives in a real-world setting. We propose AdTEC (Ad Text Evaluation Benchmark by CyberAgent), the first public benchmark to evaluate ad texts from multiple perspectives within practical advertising operations. Our contributions are as follows: (i) Defining five tasks for evaluating the quality of ad texts, as well as building a Japanese dataset based on the practical operational experiences of building a Japanese dataset based on the practical operational experiences of advertising agencies, which are typically kept in-house. (ii) Validating the performance of existing pre-trained language models (PLMs) and human evaluators on the dataset. (iii) Analyzing the characteristics and providing challenges of the benchmark. The results show that while PLMs have already reached practical usage level in several tasks, humans still outperform in certain domains, implying that there is significant room for improvement in this area.

cs.CL↗

arXiv2025

Dynamics of the intermediate-mass-element ejecta in the Supernova Remnant Cassiopeia A studied with XRISM

Supernova remnants (SNRs) provide crucial information of yet poorly understood mechanism of supernova explosion. Here we present XRISM high-resolution spectroscopy of the intermediate-mass-element (IME) ejecta in the SNR Cas A to determine their velocity distribution and thermal properties. The XRISM/Resolve spectrum in the 1.75-2.95 keV band extracted from each $1' \times 1'$ region in the southeast and northwest rims is fitted with a model consisting of two-component plasmas in non-equilibrium ionization with different radial velocities and ionization timescales. It is found that the more highly ionized component has a larger radial velocity, suggesting that this component is distributed in the outer layer and thus has been heated by the SNR reverse shock earlier. We also perform proper motion measurement of the highly ionized component (represented by the Si XIV Ly$α$ emission), using archival Chandra data, to reconstruct the three-dimensional velocity of the outermost IME ejecta. The pre-shock (free expansion) velocity of these ejecta is estimated to range from 2400 to 7100 km s$^{-1}$, based on the thermal properties and the bulk velocity of the shocked ejecta. These velocities are consistent with theoretical predictions for a Type IIb supernova, in which the progenitor's hydrogen envelope is largely stripped before the explosion. Furthermore, we find a substantial asymmetry in the distribution of the free expansion velocities, with the highest value toward the direction opposite to the proper motion of the neutron star (NS). This indicates the physical association between the asymmetric supernova explosion and NS kick.

astro-ph.HE↗

arXiv2025

Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models

As the performance of Large-scale Vision Language Models (LVLMs) improves, they are increasingly capable of responding in multiple languages, and there is an expectation that the demand for explanations generated by LVLMs will grow. However, pre-training of Vision Encoder and the integrated training of LLMs with Vision Encoder are mainly conducted using English training data, leaving it uncertain whether LVLMs can completely handle their potential when generating explanations in languages other than English. In addition, multilingual QA benchmarks that create datasets using machine translation have cultural differences and biases, remaining issues for use as evaluation tasks. To address these challenges, this study created an extended dataset in multiple languages without relying on machine translation. This dataset that takes into account nuances and country-specific phrases was then used to evaluate the generation explanation abilities of LVLMs. Furthermore, this study examined whether Instruction-Tuning in resource-rich English improves performance in other languages. Our findings indicate that LVLMs perform worse in languages other than English compared to English. In addition, it was observed that LVLMs struggle to effectively manage the knowledge learned from English data. Our dataset is available at https://huggingface.co/datasets/naist-nlp/MultiExpArt

cs.CL↗

arXiv2025

How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments

The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where the LMs are presented with a pair of acceptable and unacceptable sentences and required to judge which is more acceptable. Conventional approaches directly compare sentence probabilities assigned by LMs, but recent large language models (LLMs) are trained to perform tasks via prompting, and thus, the raw probabilities they assign may not fully reflect their grammatical knowledge. In this study, we attempt to derive more accurate acceptability judgments from LLMs using prompts and templates. Through extensive experiments in English and Chinese, we compare nine judgment methods and find two of them, a probability readout method -- in-template LP and a prompt-based method -- Yes/No probability computing, achieve higher accuracy than the conventional ones. Our analysis reveals that these methods excel in different linguistic phenomena, suggesting they access different aspects of LLMs' knowledge. We also find that ensembling the two methods outperforms single methods. Consequently, we recommend these techniques, either individually or ensembled, as more effective alternatives to conventional approaches for assessing grammatical knowledge in LLMs.

cs.CL↗

arXiv2025

Tonguescape: Exploring Language Models Understanding of Vowel Articulation

Vowels are primarily characterized by tongue position. Humans have discovered these features of vowel articulation through their own experience and explicit objective observation such as using MRI. With this knowledge and our experience, we can explain and understand the relationship between tongue positions and vowels, and this knowledge is helpful for language learners to learn pronunciation. Since language models (LMs) are trained on a large amount of data that includes linguistic and medical fields, our preliminary studies indicate that an LM is able to explain the pronunciation mechanisms of vowels. However, it is unclear whether multi-modal LMs, such as vision LMs, align textual information with visual information. One question arises: do LMs associate real tongue positions with vowel articulation? In this study, we created video and image datasets from the existing real-time MRI dataset and investigated whether LMs can understand vowel articulation based on tongue positions using vision-based information. Our findings suggest that LMs exhibit potential for understanding vowels and tongue positions when reference examples are provided while they have difficulties without them. Our code for dataset building is available on GitHub.

cs.CL↗

arXiv2025

Can Impressions of Music be Extracted from Thumbnail Images?

In recent years, there has been a notable increase in research on machine learning models for music retrieval and generation systems that are capable of taking natural language sentences as inputs. However, there is a scarcity of large-scale publicly available datasets, consisting of music data and their corresponding natural language descriptions known as music captions. In particular, non-musical information such as suitable situations for listening to a track and the emotions elicited upon listening is crucial for describing music. This type of information is underrepresented in existing music caption datasets due to the challenges associated with extracting it directly from music data. To address this issue, we propose a method for generating music caption data that incorporates non-musical aspects inferred from music thumbnail images, and validated the effectiveness of our approach through human evaluations. Additionally, we created a dataset with approximately 360,000 captions containing non-musical aspects. Leveraging this dataset, we trained a music retrieval model and demonstrated its effectiveness in music retrieval tasks through evaluation.

cs.CL↗

arXiv2024

IRR: Image Review Ranking Framework for Evaluating Vision-Language Models

Large-scale Vision-Language Models (LVLMs) process both images and text, excelling in multimodal tasks such as image captioning and description generation. However, while these models excel at generating factual content, their ability to generate and evaluate texts reflecting perspectives on the same image, depending on the context, has not been sufficiently explored. To address this, we propose IRR: Image Review Rank, a novel evaluation framework designed to assess critic review texts from multiple perspectives. IRR evaluates LVLMs by measuring how closely their judgments align with human interpretations. We validate it using a dataset of images from 15 categories, each with five critic review texts and annotated rankings in both English and Japanese, totaling over 2,000 data instances. The datasets are available at https://hf.co/datasets/naist-nlp/Wiki-ImageReview1.0. Our results indicate that, although LVLMs exhibited consistent performance across languages, their correlation with human annotations was insufficient, highlighting the need for further advancements. These findings highlight the limitations of current evaluation methods and the need for approaches that better capture human reasoning in Vision & Language tasks.

cs.CL↗

arXiv2024

Parameter estimation of protoneutron stars from gravitational wave signals using the Hilbert-Huang transform

Core-collapse supernovae (CCSNe) are potential multimessenger events detectable by current and future gravitational wave (GW) detectors. The GW signals emitted during these events are expected to provide insights into the explosion mechanism and the internal structures of neutron stars. In recent years, several studies have empirically derived the relationship between the frequencies of the GW signals originating from the oscillations of protoneutron stars (PNSs) and the physical parameters of these stars. This study applies the Hilbert-Huang transform (HHT) [Proc. R. Soc. A 454, 903 (1998)] to extract the frequencies of these modes to infer the physical properties of the PNSs. The results exhibit comparable accuracy to a short-time Fourier transform-based estimation, highlighting the potential of this approach as a complementary method for extracting physical information from GW signals of CCSNe.

gr-qc↗

arXiv2024

mbrs: A Library for Minimum Bayes Risk Decoding

Minimum Bayes risk (MBR) decoding is a decision rule of text generation tasks that outperforms conventional maximum a posterior (MAP) decoding using beam search by selecting high-quality outputs based on a utility function rather than those with high-probability. Typically, it finds the most suitable hypothesis from the set of hypotheses under the sampled pseudo-references. mbrs is a library of MBR decoding, which can flexibly combine various metrics, alternative expectation estimations, and algorithmic variants. It is designed with a focus on speed measurement and calling count of code blocks, transparency, reproducibility, and extensibility, which are essential for researchers and developers. We published our mbrs as an MIT-licensed open-source project, and the code is available on GitHub. GitHub: https://github.com/naist-nlp/mbrs

cs.CL↗