arXiv subjects

Jun Suzuki

Publications and source records attributed to Jun Suzuki.

At least 37 records · Page 2Linked to original sources

arXiv2025

Instruction-Following Evaluation of Large Vision-Language Models

Following the initial flourishing of large language models (LLMs), there has been a surge in proposed large vision-language models (LVLMs) that integrate LLMs with vision capabilities. However, it has been observed that LVLMs, after tuning to visual instruction using commonly used training datasets, often fail to exhibit the instruction-following ability that was present in the LLM before integration, leading to results in which they do not follow task instructions as expected. This study quantitatively demonstrates that LVLMs' instruction-following ability declines after fine-tuning and analyzes its underlying causes. In particular, we constructed new training datasets highlighting whether the output format is specified. Then, we investigated how explicitly indicating the output format during fine-tuning affects LVLMs' instruction-following ability. Our quantitative evaluation confirmed that LVLMs' instruction-following ability declines after fine-tuning with commonly used datasets. Furthermore, we found that LVLMs trained with datasets, including instructions on output format, tend to follow instructions more accurately than models that do not. These findings suggest that including samples with instructions on output format during (visual) instruction tuning may help mitigate the decline in instruction-following abilities.

cs.CL↗

arXiv2025

An Open and Reproducible Deep Research Agent for Long-Form Question Answering

We present an open deep research system for long-form question answering, selected as a winning system in the text-to-text track of the MMU-RAG competition at NeurIPS 2025. The system combines an open-source large language model (LLM) with an open web search API to perform iterative retrieval, reasoning, and synthesis in real-world open-domain settings. To enhance reasoning quality, we apply preference tuning based on LLM-as-a-judge feedback that evaluates multiple aspects, including clarity, insightfulness, and factuality. Our experimental results show that the proposed method consistently improves answer quality across all three aspects. Our source code is publicly available at https://github.com/efficient-deep-research/efficient-deep-research.

cs.CL↗

arXiv2025

Saturating the Quantum Cramér--Rao Bound in Prioritised Parameter Estimation

Measurement incompatibility is a cornerstone of quantum mechanics. In the context of estimating multiple parameters of a quantum system, this manifests as a fundamental trade-off between the precisions with which different parameters can be estimated. Often, a parameter can be optimally measured, but at the cost of gaining no information about incompatible parameters. Here, we report that there are systems where one parameter's information can be maximised while not completely losing information about the other parameters. In doing so, we find attainable trade-off relations for quantum parameter estimation with a structure that is different to typical Heisenberg-type trade-offs. We demonstrate our findings by implementing an optimal entangling measurement on a Quantinuum trapped-ion quantum computer.

quant-ph↗

arXiv2025

Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders

Recent interpretability work on large language models (LLMs) has been increasingly dominated by a feature-discovery approach with the help of proxy modules. Then, the quality of features learned by, e.g., sparse auto-encoders (SAEs), is evaluated. This paradigm naturally raises a critical question: do such learned features have better properties than those already represented within the original model parameters, and unfortunately, only a few studies have made such comparisons systematically so far. In this work, we revisit the interpretability of feature vectors stored in feed-forward (FF) layers, given the perspective of FF as key-value memories, with modern interpretability benchmarks. Our extensive evaluation revealed that SAE and FFs exhibits a similar range of interpretability, although SAEs displayed an observable but minimal improvement in some aspects. Furthermore, in certain aspects, surprisingly, even vanilla FFs yielded better interpretability than the SAEs, and features discovered in SAEs and FFs diverged. These bring questions about the advantage of SAEs from both perspectives of feature quality and faithfulness, compared to directly interpreting FF feature vectors, and FF key-value parameters serve as a strong baseline in modern interpretability research.

cs.LG↗

arXiv2025

Hybrid Cramér-Rao bound for Quantum Bayes-Point Estimation with Nuisance Parameters

We develop a hybrid framework for quantum parameter estimation in the presence of nuisance parameters. In this Bayes-point scheme, the parameters of interest are treated as fixed non-random parameters while nuisance parameters are integrated out with respect to a prior (random parameters). Within this setting, we introduce the hybrid partial quantum Fisher information matrix (hpQFIM), defined by prior-averaging the nuisance block of the QFIM and taking a Schur complement, and derive a corresponding Cramér-Rao-type lower bound on the hybrid risk. We establish structural properties of the hpQFIM, including inequalities that bracket it between computationally tractable surrogates, as well as limiting behaviors under extreme priors. Operationally, the hybrid approach improves over pure point estimation since the optimal measurement for the parameters of interest depends only on the prior distribution of the nuisance, rather than on its unknown value. We illustrate the framework with analytically solvable qubit models and numerical examples, clarifying how partial prior information on nuisance variables can be systematically exploited in quantum metrology.

quant-ph↗

arXiv2025

Holevo Cramér-Rao bound: How close can we get without entangling measurements?

In multi-parameter quantum metrology, the resource of entanglement can lead to an increase in efficiency of the estimation process. Entanglement can be used in the state preparation stage, or the measurement stage, or both, to harness this advantage; here we focus on the role of entangling measurements. Specifically, entangling or collective measurements over multiple identical copies of a probe state are known to be superior to measuring each probe individually, but the extent of this improvement is an open problem. It is also known that such entangling measurements, though resource-intensive, are required to attain the ultimate limits in multi-parameter quantum metrology and quantum information processing tasks. In this work we investigate the maximum precision improvement that collective quantum measurements can offer over individual measurements for estimating parameters of qudit states, calling this the 'collective quantum enhancement'. We show that, whereas the maximum enhancement can, in principle, be a factor of $n$ for estimating $n$ parameters, this bound is not tight for large $n$. Instead, our results prove an enhancement linear in dimension of the qudit is possible using collective measurements and lead us to conjecture that this is the maximum collective quantum enhancement in any local estimation scenario.

quant-ph↗

arXiv2025

Undecidability of Linear Logics without Weakening

The goal of this paper is to establish that it remains undecidable whether a sequent is provable in two systems in which a weakening rule for an exponential modality is completely omitted from classical propositional linear logic $\mathbf{CLL}$ introduced by Girard (1987), which is shown to be undecidable by Lincoln et al. (1992). We introduce two logical systems, $\mathbf{CLLR}$ and $\mathbf{CLLRR}$. The first system, $\mathbf{CLLR}$, is obtained by omitting the weakening rule for the exponential modality of $\mathbf{CLL}$. The system $\mathbf{CLLR}$ has been studied by several authors, including Meliès-Tabareau (2010), but its undecidability was unknown. This paper shows the undecidability of $\mathbf{CLLR}$ by reducing it to the undecidability of $\mathbf{CLL}$, where the units $\mathbf{1}$ and $\bot$ play a crucial role in simulating the weakening rule. We also omit these units from the syntax and inference rules of $\mathbf{CLLR}$ in order to define the second system, $\mathbf{CLLRR}$. The undecidability of $\mathbf{CLLRR}$ is established by showing that the system can simulate any two-counter machine proposed by Minsky (1961).

cs.LO↗

arXiv2025

Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models

This study investigates the layerwise importance of feed-forward networks (FFNs) in Transformer-based language models during pretraining. We introduce an experimental approach that, while maintaining the total parameter count, increases the FFN dimensions in some layers and completely removes the FFNs from other layers. Furthermore, since our focus is on the importance of FFNs during pretraining, we train models from scratch to examine whether the importance of FFNs varies depending on their layer positions, rather than using publicly available pretrained models as is frequently done. Through comprehensive evaluations of models with varying sizes (285M, 570M, and 1.2B parameters) and layer counts (12, 24, and 40 layers), we demonstrate that concentrating FFNs in 70% of the consecutive middle layers consistently outperforms standard configurations for multiple downstream tasks.

cs.CL↗

arXiv2025

Spike No More: Stabilizing the Pre-training of Large Language Models

Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre-training needs a vast computational budget, we should avoid such spikes. Based on the assumption that the loss spike is caused by the sudden growth of the gradient norm, we explore factors to keep the gradient norm small through an analysis of the spectral norms of the Jacobian matrices for the sub-layers. Our findings suggest that stabilizing the pre-training process requires two conditions: small sub-layers and large shortcut. We conduct various experiments to empirically verify our theoretical analyses. Experimental results demonstrate that methods satisfying the conditions effectively prevent loss spikes during pre-training.

cs.CL↗

arXiv2025

How Stylistic Similarity Shapes Preferences in Dialogue Dataset with User and Third Party Evaluations

Recent advancements in dialogue generation have broadened the scope of human-bot interactions, enabling not only contextually appropriate responses but also the analysis of human affect and sensitivity. While prior work has suggested that stylistic similarity between user and system may enhance user impressions, the distinction between subjective and objective similarity is often overlooked. To investigate this issue, we introduce a novel dataset that includes users' preferences, subjective stylistic similarity based on users' own perceptions, and objective stylistic similarity annotated by third party evaluators in open-domain dialogue settings. Analysis using the constructed dataset reveals a strong positive correlation between subjective stylistic similarity and user preference. Furthermore, our analysis suggests an important finding: users' subjective stylistic similarity differs from third party objective similarity. This underscores the importance of distinguishing between subjective and objective evaluations and understanding the distinct aspects each captures when analyzing the relationship between stylistic similarity and user preferences. The dataset presented in this paper is available online.

cs.CL↗

arXiv2025

Can Input Attributions Explain Inductive Reasoning in In-Context Learning?

Interpreting the internal process of neural models has long been a challenge. This challenge remains relevant in the era of large language models (LLMs) and in-context learning (ICL); for example, ICL poses a new issue of interpreting which example in the few-shot examples contributed to identifying/solving the task. To this end, in this paper, we design synthetic diagnostic tasks of inductive reasoning, inspired by the generalization tests typically adopted in psycholinguistics. Here, most in-context examples are ambiguous w.r.t. their underlying rule, and one critical example disambiguates it. The question is whether conventional input attribution (IA) methods can track such a reasoning process, i.e., identify the influential example, in ICL. Our experiments provide several practical findings; for example, a certain simple IA method works the best, and the larger the model, the generally harder it is to interpret the ICL with gradient-based IA methods.

cs.CL↗

arXiv2025

VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents

We aim to develop a retrieval-augmented generation (RAG) framework that answers questions over a corpus of visually-rich documents presented in mixed modalities (e.g., charts, tables) and diverse formats (e.g., PDF, PPTX). In this paper, we introduce a new RAG framework, VDocRAG, which can directly understand varied documents and modalities in a unified image format to prevent missing information that occurs by parsing documents to obtain text. To improve the performance, we propose novel self-supervised pre-training tasks that adapt large vision-language models for retrieval by compressing visual information into dense token representations while aligning them with textual content in documents. Furthermore, we introduce OpenDocVQA, the first unified collection of open-domain document visual question answering datasets, encompassing diverse document types and formats. OpenDocVQA provides a comprehensive resource for training and evaluating retrieval and question answering models on visually-rich documents in an open-domain setting. Experiments show that VDocRAG substantially outperforms conventional text-based RAG and has strong generalization capability, highlighting the potential of an effective RAG paradigm for real-world documents.

cs.CL↗

arXiv2025

STEP: Staged Parameter-Efficient Pre-training for Large Language Models

Pre-training large language models (LLMs) faces significant memory challenges due to the large size of model parameters. We introduce STaged parameter-Efficient Pre-training (STEP), which integrates parameter-efficient tuning techniques with model growth. We conduct experiments on pre-training LLMs of various sizes and demonstrate that STEP achieves up to a 53.9% reduction in maximum memory requirements compared to vanilla pre-training while maintaining equivalent performance. Furthermore, we show that the model by STEP performs comparably to vanilla pre-trained models on downstream tasks after instruction tuning.

cs.CL↗

arXiv2025

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization

The Mixture of Experts (MoE) architecture reduces the training and inference cost significantly compared to a dense model of equivalent capacity. Upcycling is an approach that initializes and trains an MoE model using a pre-trained dense model. While upcycling leads to initial performance gains, the training progresses slower than when trained from scratch, leading to suboptimal performance in the long term. We propose Drop-Upcycling - a method that effectively addresses this problem. Drop-Upcycling combines two seemingly contradictory approaches: utilizing the knowledge of pre-trained dense models while statistically re-initializing some parts of the weights. This approach strategically promotes expert specialization, significantly enhancing the MoE model's efficiency in knowledge acquisition. Extensive large-scale experiments demonstrate that Drop-Upcycling significantly outperforms previous MoE construction methods in the long term, specifically when training on hundreds of billions of tokens or more. As a result, our MoE model with 5.9B active parameters achieves comparable performance to a 13B dense model in the same model family, while requiring approximately 1/4 of the training FLOPs. All experimental resources, including source code, training data, model checkpoints and logs, are publicly available to promote reproducibility and future research on MoE.

cs.CL↗

arXiv2025

MQM-Chat: Multidimensional Quality Metrics for Chat Translation

The complexities of chats pose significant challenges for machine translation models. Recognizing the need for a precise evaluation metric to address the issues of chat translation, this study introduces Multidimensional Quality Metrics for Chat Translation (MQM-Chat). Through the experiments of five models using MQM-Chat, we observed that all models generated certain fundamental errors, while each of them has different shortcomings, such as omission, overly correcting ambiguous source content, and buzzword issues, resulting in the loss of stylized information. Our findings underscore the effectiveness of MQM-Chat in evaluating chat translation, emphasizing the importance of stylized content and dialogue consistency for future studies.

cs.CL↗

arXiv2025

Reference-free Evaluation Metrics for Text Generation: A Survey

A number of automatic evaluation metrics have been proposed for natural language generation systems. The most common approach to automatic evaluation is the use of a reference-based metric that compares the model's output with gold-standard references written by humans. However, it is expensive to create such references, and for some tasks, such as response generation in dialogue, creating references is not a simple matter. Therefore, various reference-free metrics have been developed in recent years. In this survey, which intends to cover the full breadth of all NLG tasks, we investigate the most commonly used approaches, their application, and their other uses beyond evaluating models. The survey concludes by highlighting some promising directions for future research.

cs.CL↗

arXiv2024

LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs

This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and strong Japanese LLMs, and as of this writing, more than 1,500 participants from academia and industry are working together for this purpose. This paper presents the background of the establishment of LLM-jp, summaries of its activities, and technical reports on the LLMs developed by LLM-jp. For the latest activities, visit https://llm-jp.nii.ac.jp/en/.

cs.CL↗

arXiv2024

An Investigation of Warning Erroneous Chat Translations in Cross-lingual Communication

Machine translation models are still inappropriate for translating chats, despite the popularity of translation software and plug-in applications. The complexity of dialogues poses significant challenges and can hinder crosslingual communication. Instead of pursuing a flawless translation system, a more practical approach would be to issue warning messages about potential mistranslations to reduce confusion. However, it is still unclear how individuals perceive these warning messages and whether they benefit the crowd. This paper tackles to investigate this question and demonstrates the warning messages' contribution to making chat translation systems effective.

cs.CL↗