arXiv subjects

Jun Suzuki

Publications and source records attributed to Jun Suzuki.

At least 55 records · Page 3Linked to original sources

arXiv2024

Pruning Multilingual Large Language Models for Multilingual Inference

Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. A distinctive characteristic of MLLMs is their high-quality translation capabilities, indicating an acquired proficiency in aligning between languages. This study explores how to enhance the zero-shot performance of MLLMs in non-English languages by leveraging their alignment capability between English and non-English languages. To achieve this, we first analyze the behavior of MLLMs when performing translation and reveal that there are large magnitude features that play a critical role in the translation process. Inspired by these findings, we retain the weights associated with operations involving the large magnitude features and prune other weights to force MLLMs to rely on these features for tasks beyond translation. We empirically demonstrate that this pruning strategy can enhance the MLLMs' performance in non-English language.

cs.CL↗

arXiv2024

The gap persistence theorem for quantum multiparameter estimation

One key aspect of quantum metrology, measurement incompatibility, is evident only through the simultaneous estimation of multiple parameters. The symmetric logarithmic derivative Cramér-Rao bound (SLDCRB), gives the attainable precision, if the optimal measurements for estimating each individual parameter commute. When the optimal measurements do not commute, the SLDCRB is not necessarily attainable. In this regard, the Holevo Cramér-Rao bound (HCRB) plays a fundamental role, providing the ultimate attainable precisions when one allows simultaneous measurements on infinitely many copies of a quantum state. For practical purposes, the Nagaoka Cramér-Rao bound (NCRB) is more relevant, applying when restricted to measuring quantum states individually. The interplay between these three bounds dictates how rapidly the ultimate metrological precisions can be approached through collective measurements on finite copies of the probe state. We first consider two parameter estimation and prove that if the HCRB cannot be saturated with a single copy of the probe state, then it cannot be saturated for any finite number of copies of the probe state. With this, we show that it is impossible to saturate the HCRB for several physically motivated problems. For estimating any number of parameters, we provide necessary and sufficient conditions for the attainability of the SLDCRB with separable measurements. We further prove that if the SLDCRB cannot be reached with a single copy of the probe state, it cannot be reached with collective measurements on any finite number of copies of the probe state. These results together provide necessary and sufficient conditions for the attainability of the SLDCRB for any finite number of copies of the probe state. This solves a significant generalisation of one of the five problems recently highlighted by [P.Horodecki et al, Phys. Rev. X Quantum 3, 010101 (2022)].

quant-ph↗

arXiv2024

Bayesian Logarithmic Derivative Type Lower Bounds for Quantum Estimation

Bayesian approach for quantum parameter estimation has gained a renewed interest from practical applications of quantum estimation theory. Recently, a lower bound, called the Bayesian Nagaoka-Hayashi bound for the Bayes risk in quantum domain was proposed, which is an extension of a new approach to point estimation of quantum states by Conlon et al. (2021). The objective of this paper is to explore this Bayesian Nagaoka-Hayashi bound further by obtaining its lower bounds. We first obtain one-parameter family of lower bounds, which is an analogue of the Holevo bound in point estimation. Thereby, we derive one-parameter family of Bayesian logarithmic derivative type lower bounds in a closed form for the parameter independent weight matrix setting. This new bound includes previously known Bayesian lower bounds as special cases.

quant-ph↗

arXiv2024

Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection

We explore visual prompt injection (VPI) that maliciously exploits the ability of large vision-language models (LVLMs) to follow instructions drawn onto the input image. We propose a new VPI method, "goal hijacking via visual prompt injection" (GHVPI), that swaps the execution task of LVLMs from an original task to an alternative task designated by an attacker. The quantitative analysis indicates that GPT-4V is vulnerable to the GHVPI and demonstrates a notable attack success rate of 15.8%, which is an unignorable security risk. Our analysis also shows that successful GHVPI requires high character recognition capability and instruction-following ability in LVLMs.

cs.CL↗

arXiv2024

QestOptPOVM: An iterative algorithm to find optimal measurements for quantum parameter estimation

Quantum parameter estimation holds significant promise for achieving high precision through the utilization of the most informative measurements. While various lower bounds have been developed to assess the best accuracy for estimates, they are not tight, nor provide a construction of the optimal measurement in general. Thus, determining the explicit forms of optimal measurements has been challenging due to the non-trivial optimization. In this study, we introduce an algorithm, termed QestOptPOVM, designed to directly identify optimal positive operator-valued measure (POVM) using the steepest descent method. Through rigorous testing on several examples for multiple copies of qubit states (up to six copies), we demonstrate the efficiency and accuracy of our proposed algorithm. Moreover, a comparative analysis between numerical results and established lower bounds serves to validate the tightness of the Nagaoka-Hayashi bound in finite-sample quantum metrology for our examples. Concurrently, our algorithm functions as a tool for elucidating the explicit forms of optimal POVMs, thereby enhancing our understanding of quantum parameter estimation methodologies.

quant-ph↗

arXiv2024

Detecting Response Generation Not Requiring Factual Judgment

With the remarkable development of large language models (LLMs), ensuring the factuality of output has become a challenge. However, having all the contents of the response with given knowledge or facts is not necessarily a good thing in dialogues. This study aimed to achieve both attractiveness and factuality in a dialogue response for which a task was set to predict sentences that do not require factual correctness judgment such as agreeing, or personal opinions/feelings. We created a dataset, dialogue dataset annotated with fact-check-needed label (DDFC), for this task via crowdsourcing, and classification tasks were performed on several models using this dataset. The model with the highest classification accuracy could yield about 88% accurate classification results.

cs.CL↗

arXiv2024

Role of the extended Hilbert space in the attainability of the Quantum Cramér-Rao bound for multiparameter estimation

The symmetric logarithmic derivative Cramér-Rao bound (SLDCRB) provides a fundamental limit to the minimum variance with which a set of unknown parameters can be estimated in an unbiased manner. It is known that the SLDCRB can be saturated provided the optimal measurements for the individual parameters commute with one another. However, when this is not the case the SLDCRB cannot be attained in general. In the experimentally relevant setting, where quantum states are measured individually, necessary and sufficient conditions for when the SLDCRB can be saturated are not known. In this setting the SLDCRB is attainable provided the SLD operators can be chosen to commute on an extended Hilbert space. However, beyond this relatively little is known about when the SLD operators can be chosen in this manner. In this paper we present explicit examples which demonstrate novel aspects of this condition. Our examples demonstrate that the SLD operators commuting on any two of the following three spaces: support space, support-kernel space and kernel space, is neither a necessary nor sufficient condition for commutativity on the extended space. We present a simple analytic example showing that the Nagaoka-Hayashi Cramér-Rao bound is not always attainable. Finally, we provide necessary and sufficient conditions for the attainability of the SLDCRB in the case when the kernel space is one-dimensional. These results provide new information on the necessary and sufficient conditions for the attainability of the SLDCRB.

quant-ph↗

arXiv2024

A Large Collection of Model-generated Contradictory Responses for Consistency-aware Dialogue Systems

Mitigating the generation of contradictory responses poses a substantial challenge in dialogue response generation. The quality and quantity of available contradictory response data play a vital role in suppressing these contradictions, offering two significant benefits. First, having access to large contradiction data enables a comprehensive examination of their characteristics. Second, data-driven methods to mitigate contradictions may be enhanced with large-scale contradiction data for training. Nevertheless, no attempt has been made to build an extensive collection of model-generated contradictory responses. In this paper, we build a large dataset of response generation models' contradictions for the first time. Then, we acquire valuable insights into the characteristics of model-generated contradictions through an extensive analysis of the collected responses. Lastly, we also demonstrate how this dataset substantially enhances the performance of data-driven contradiction suppression methods.

cs.CL↗

arXiv2024

InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions

We study the problem of completing various visual document understanding (VDU) tasks, e.g., question answering and information extraction, on real-world documents through human-written instructions. To this end, we propose InstructDoc, the first large-scale collection of 30 publicly available VDU datasets, each with diverse instructions in a unified format, which covers a wide range of 12 tasks and includes open document types/formats. Furthermore, to enhance the generalization performance on VDU tasks, we design a new instruction-based document reading and understanding model, InstructDr, that connects document images, image encoders, and large language models (LLMs) through a trainable bridging module. Experiments demonstrate that InstructDr can effectively adapt to new VDU datasets, tasks, and domains via given instructions and outperforms existing multimodal LLMs and ChatGPT without specific training.

cs.CV↗

arXiv2023

Error tradeoff relation for estimating the unitary-shift parameter of a relativistic spin-1/2 particle

The purpose of this paper is to discuss the existence of a nontrivial tradeoff relation for estimating two unitary-shift parameters in a relativistic spin-1/2 system. It is shown that any moving observer cannot estimate two parameters simultaneously, even though a parametric model is classical in the rest frame. This transition from the classical model to a genuine quantum model is investigated analytically using a one-parameter family of quantum Fisher information matrices. This paper proposes to use an indicator that can not only detect the existence of a tradeoff relation but can also evaluate its strength. Based on the proposed indicator, this paper investigates the nature of the tradeoff relation in detail.

quant-ph↗

arXiv2023

A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video

This paper proposes a practical multimodal video summarization task setting and a dataset to train and evaluate the task. The target task involves summarizing a given video into a predefined number of keyframe-caption pairs and displaying them in a listable format to grasp the video content quickly. This task aims to extract crucial scenes from the video in the form of images (keyframes) and generate corresponding captions explaining each keyframe's situation. This task is useful as a practical application and presents a highly challenging problem worthy of study. Specifically, achieving simultaneous optimization of the keyframe selection performance and caption quality necessitates careful consideration of the mutual dependence on both preceding and subsequent keyframes and captions. To facilitate subsequent research in this field, we also construct a dataset by expanding upon existing datasets and propose an evaluation framework. Furthermore, we develop two baseline systems and report their respective performance.

cs.CL↗

arXiv2023

Refactoring Programs Using Large Language Models with Few-Shot Examples

A less complex and more straightforward program is a crucial factor that enhances its maintainability and makes writing secure and bug-free programs easier. However, due to its heavy workload and the risks of breaking the working programs, programmers are reluctant to do code refactoring, and thus, it also causes the loss of potential learning experiences. To mitigate this, we demonstrate the application of using a large language model (LLM), GPT-3.5, to suggest less complex versions of the user-written Python program, aiming to encourage users to learn how to write better programs. We propose a method to leverage the prompting with few-shot examples of the LLM by selecting the best-suited code refactoring examples for each target programming problem based on the prior evaluation of prompting with the one-shot example. The quantitative evaluation shows that 95.68% of programs can be refactored by generating 10 candidates each, resulting in a 17.35% reduction in the average cyclomatic complexity and a 25.84% decrease in the average number of lines after filtering only generated programs that are semantically correct. Furthermore, the qualitative evaluation shows outstanding capability in code formatting, while unnecessary behaviors such as deleting or translating comments are also observed.

cs.PL↗

arXiv2023

Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism

Large language models (LLMs) take advantage of step-by-step reasoning instructions, e.g., chain-of-thought (CoT) prompting. Building on this, their ability to perform CoT-style reasoning robustly is of interest from a probing perspective. In this study, we inspect the step-by-step reasoning ability of LLMs with a focus on negation, which is a core linguistic phenomenon that is difficult to process. In particular, we introduce several controlled settings (e.g., reasoning in case of fictional entities) to evaluate the logical reasoning abilities of the models. We observed that dozens of modern LLMs were not robust against lexical negation (e.g., plausible ->implausible) when performing CoT-style reasoning, and the results highlight unique limitations in each LLM family.

cs.CL↗

arXiv2023

Chat Translation Error Detection for Assisting Cross-lingual Communications

In this paper, we describe the development of a communication support system that detects erroneous translations to facilitate crosslingual communications due to the limitations of current machine chat translation methods. We trained an error detector as the baseline of the system and constructed a new Japanese-English bilingual chat corpus, BPersona-chat, which comprises multiturn colloquial chats augmented with crowdsourced quality ratings. The error detector can serve as an encouraging foundation for more advanced erroneous translation detection systems.

cs.CL↗

arXiv2023

Bayesian Nagaoka-Hayashi Bound for Multiparameter Quantum-State Estimation Problem

In this work we propose a Bayesian version of the Nagaoka-Hayashi bound when estimating a parametric family of quantum states. This lower bound is a generalization of a recently proposed bound for point estimation to Bayesian estimation. We then show that the proposed lower bound can be efficiently computed as a semidefinite programming problem. As a lower bound, we also derive a Bayesian version of the Holevo-type bound from the Bayesian Nagaoka-Hayashi bound. Lastly, we prove that the new lower bound is tighter than the Bayesian quantum Cramer-Rao bounds.

quant-ph↗

arXiv2023

Exploring the Robustness of Large Language Models for Solving Programming Problems

Using large language models (LLMs) for source code has recently gained attention. LLMs, such as Transformer-based models like Codex and ChatGPT, have been shown to be highly capable of solving a wide range of programming problems. However, the extent to which LLMs understand problem descriptions and generate programs accordingly or just retrieve source code from the most relevant problem in training data based on superficial cues has not been discovered yet. To explore this research question, we conduct experiments to understand the robustness of several popular LLMs, CodeGen and GPT-3.5 series models, capable of tackling code generation tasks in introductory programming problems. Our experimental results show that CodeGen and Codex are sensitive to the superficial modifications of problem descriptions and significantly impact code generation performance. Furthermore, we observe that Codex relies on variable names, as randomized variables decrease the solved rate significantly. However, the state-of-the-art (SOTA) models, such as InstructGPT and ChatGPT, show higher robustness to superficial modifications and have an outstanding capability for solving programming problems. This highlights the fact that slight modifications to the prompts given to the LLMs can greatly affect code generation performance, and careful formatting of prompts is essential for high-quality code generation, while the SOTA models are becoming more robust to perturbations.

cs.CL↗

arXiv2023

B2T Connection: Serving Stability and Performance in Deep Transformers

From the perspective of the layer normalization (LN) positions, the architectures of Transformers can be categorized into two types: Post-LN and Pre-LN. Recent Transformers tend to be Pre-LN because, in Post-LN with deep Transformers (e.g., those with ten or more layers), the training is often unstable, resulting in useless models. However, Post-LN has consistently achieved better performance than Pre-LN in relatively shallow Transformers (e.g., those with six or fewer layers). This study first investigates the reason for these discrepant observations empirically and theoretically and made the following discoveries: 1, the LN in Post-LN is the main source of the vanishing gradient problem that leads to unstable training, whereas Pre-LN prevents it, and 2, Post-LN tends to preserve larger gradient norms in higher layers during the back-propagation, which may lead to effective training. Exploiting the new findings, we propose a method that can provide both high stability and effective training by a simple modification of Post-LN. We conduct experiments on a wide range of text generation tasks. The experimental results demonstrate that our method outperforms Pre-LN, and enables stable training regardless of the shallow or deep layer settings. Our code is publicly available at https://github.com/takase/b2t_connection.

cs.LG↗

arXiv2022

Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems

Automation of dialogue system evaluation is a driving force for the efficient development of dialogue systems. This paper introduces the bipartite-play method, a dialogue collection method for automating dialogue system evaluation. It addresses the limitations of existing dialogue collection methods: (i) inability to compare with systems that are not publicly available, and (ii) vulnerability to cheating by intentionally selecting systems to be compared. Experimental results show that the automatic evaluation using the bipartite-play method mitigates these two drawbacks and correlates as strongly with human subjectivity as existing methods.

cs.CL↗