arXiv subjects

Matteo Negri

Publications and source records attributed to Matteo Negri.

At least 37 records · Page 2Linked to original sources

arXiv2025

The Unheard Alternative: Contrastive Explanations for Speech-to-Text Models

Contrastive explanations, which indicate why an AI system produced one output (the target) instead of another (the foil), are widely regarded in explainable AI as more informative and interpretable than standard explanations. However, obtaining such explanations for speech-to-text (S2T) generative models remains an open challenge. Drawing from feature attribution techniques, we propose the first method to obtain contrastive explanations in S2T by analyzing how parts of the input spectrogram influence the choice between alternative outputs. Through a case study on gender assignment in speech translation, we show that our method accurately identifies the audio features that drive the selection of one gender over another. By extending the scope of contrastive explanations to S2T, our work provides a foundation for better understanding S2T models.

cs.CL↗

arXiv2025

Translation in the Hands of Many:Centering Lay Users in Machine Translation Interactions

Converging societal and technical factors have transformed language technologies into user-facing applications used by the general public across languages. Machine Translation (MT) has become a global tool, with cross-lingual services now also supported by dialogue systems powered by multilingual Large Language Models (LLMs). Widespread accessibility has extended MT's reach to a vast base of lay users, many with little to no expertise in the languages or the technology itself. And yet, the understanding of MT consumed by such a diverse group of users -- their needs, experiences, and interactions with multilingual systems -- remains limited. In our position paper, we first trace the evolution of MT user profiles, focusing on non-experts and how their engagement with technology may shift with the rise of LLMs. Building on an interdisciplinary body of work, we identify three factors -- usability, trust, and literacy -- that are central to shaping user interactions and must be addressed to align MT with user needs. By examining these dimensions, we provide insights to guide the progress of more user-centered MT.

cs.CL↗

arXiv2025

Mind the Inclusivity Gap: Multilingual Gender-Neutral Translation Evaluation with mGeNTE

Avoiding the propagation of undue (binary) gender inferences and default masculine language remains a key challenge towards inclusive multilingual technologies, particularly when translating into languages with extensive gendered morphology. Gender-neutral translation (GNT) represents a linguistic strategy towards fairer communication across languages. However, research on GNT is limited to a few resources and language pairs. To address this gap, we introduce mGeNTE, an expert-curated resource, and use it to conduct the first systematic multilingual evaluation of inclusive translation with state-of-the-art instruction-following language models (LMs). Experiments on en-es/de/it/el reveal that while models can recognize when neutrality is appropriate, they cannot consistently produce neutral translations, limiting their usability. To probe this behavior, we enrich our evaluation with interpretability analyses that identify task-relevant features and offer initial insights into the internal dynamics of LM-based GNT.

cs.CL↗

arXiv2025

Gender-Neutral Rewriting in Italian: Models, Approaches, and Trade-offs

Gender-neutral rewriting (GNR) aims to reformulate text to eliminate unnecessary gender specifications while preserving meaning, a particularly challenging task in grammatical-gender languages like Italian. In this work, we conduct the first systematic evaluation of state-of-the-art large language models (LLMs) for Italian GNR, introducing a two-dimensional framework that measures both neutrality and semantic fidelity to the input. We compare few-shot prompting across multiple LLMs, fine-tune selected models, and apply targeted cleaning to boost task relevance. Our findings show that open-weight LLMs outperform the only existing model dedicated to GNR in Italian, whereas our fine-tuned models match or exceed the best open-weight LLM's performance at a fraction of its size. Finally, we discuss the trade-off between optimizing the training data for neutrality and meaning preservation.

cs.CL↗

arXiv2025

Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings

Energy-based probabilistic models learned by maximizing the likelihood of the data are limited by the intractability of the partition function. A widely used workaround is to maximize the pseudo-likelihood, which replaces the global normalization with tractable local normalizations. Here we show that, in the zero-temperature limit, a network trained to maximize pseudo-likelihood naturally implements an associative memory: if the training set is small, patterns become fixed-point attractors whose basins of attraction exceed those of any classical Hopfield rule. We explain quantitatively this effect on uncorrelated random patterns. Moreover, we show that, for different structured datasets coming from computer science (random feature model, MNIST), physics (spin glasses) and biology (proteins), as the number of training examples increases the learned network goes beyond memorization, developing meaningful attractors with non-trivial correlations with test examples, thus showing the ability to generalize. Our results therefore reveal pseudo-likelihood works both as an efficient inference tool and as a principled mechanism for memory and generalization.

cond-mat.stat-mech↗

arXiv2025

Different Speech Translation Models Encode and Translate Speaker Gender Differently

Recent studies on interpreting the hidden states of speech models have shown their ability to capture speaker-specific features, including gender. Does this finding also hold for speech translation (ST) models? If so, what are the implications for the speaker's gender assignment in translation? We address these questions from an interpretability perspective, using probing methods to assess gender encoding across diverse ST models. Results on three language directions (English-French/Italian/Spanish) indicate that while traditional encoder-decoder models capture gender information, newer architectures -- integrating a speech encoder with a machine translation system via adapters -- do not. We also demonstrate that low gender encoding capabilities result in systems' tendency toward a masculine default, a translation bias that is more pronounced in newer architectures.

cs.CL↗

arXiv2025

Echoes of Phonetics: Unveiling Relevant Acoustic Cues for ASR via Feature Attribution

Despite significant advances in ASR, the specific acoustic cues models rely on remain unclear. Prior studies have examined such cues on a limited set of phonemes and outdated models. In this work, we apply a feature attribution technique to identify the relevant acoustic cues for a modern Conformer-based ASR system. By analyzing plosives, fricatives, and vowels, we assess how feature attributions align with their acoustic properties in the time and frequency domains, also essential for human speech perception. Our findings show that the ASR model relies on vowels' full time spans, particularly their first two formants, with greater saliency in male speech. It also better captures the spectral characteristics of sibilant fricatives than non-sibilants and prioritizes the release phase in plosives, especially burst characteristics. These insights enhance the interpretability of ASR models and highlight areas for future research to uncover potential gaps in model robustness.

cs.CL↗

arXiv2025

FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian

The development of speech foundation models (SFMs) like Whisper and SeamlessM4T has significantly advanced the field of speech processing. However, their closed nature--with inaccessible training data and code--poses major reproducibility and fair evaluation challenges. While other domains have made substantial progress toward open science by developing fully transparent models trained on open-source (OS) code and data, similar efforts in speech remain limited. To fill this gap, we introduce FAMA, the first family of open science SFMs for English and Italian, trained on 150k+ hours of OS speech data. Moreover, we present a new dataset containing 16k hours of cleaned and pseudo-labeled speech for both languages. Results show that FAMA achieves competitive performance compared to existing SFMs while being up to 8 times faster. All artifacts, including code, datasets, and models, are released under OS-compliant licenses, promoting openness in speech technology research.

cs.CL↗

arXiv2025

The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence

Training large-scale models presents challenges not only in terms of resource requirements but also in terms of their convergence. For this reason, the learning rate (LR) is often decreased when the size of a model is increased. Such a simple solution is not enough in the case of speech-to-text (S2T) trainings, where evolved and more complex variants of the Transformer architecture -- e.g., Conformer or Branchformer -- are used in light of their better performance. As a workaround, OWSM designed a double linear warmup of the LR, increasing it to a very small value in the first phase before updating it to a higher value in the second phase. While this solution worked well in practice, it was not compared with alternative solutions, nor was the impact on the final performance of different LR warmup schedules studied. This paper fills this gap, revealing that i) large-scale S2T trainings demand a sub-exponential LR warmup, and ii) a higher LR in the warmup phase accelerates initial convergence, but it does not boost final performance.

cs.CL↗

arXiv2025

An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation

Gender-neutral translation (GNT) aims to avoid expressing the gender of human referents when the source text lacks explicit cues about the gender of those referents. Evaluating GNT automatically is particularly challenging, with current solutions being limited to monolingual classifiers. Such solutions are not ideal because they do not factor in the source sentence and require dedicated data and fine-tuning to scale to new languages. In this work, we address such limitations by investigating the use of large language models (LLMs) as evaluators of GNT. Specifically, we explore two prompting approaches: one in which LLMs generate sentence-level assessments only, and another, akin to a chain-of-thought approach, where they first produce detailed phrase-level annotations before a sentence-level judgment. Through extensive experiments on multiple languages with five models, both open and proprietary, we show that LLMs can serve as evaluators of GNT. Moreover, we find that prompting for phrase-level annotations before sentence-level assessments consistently improves the accuracy of all models, providing a better and more scalable alternative to current solutions.

cs.CL↗

arXiv2025

Low-power multi-mode fiber projector overcomes shallow neural networks classifiers

In the domain of disordered photonics, the characterization of optically opaque materials for light manipulation and imaging is a primary aim. Among various complex devices, multi-mode optical fibers stand out as cost-effective and easy-to-handle tools, making them attractive for several tasks. In this context, we cast these fibers into random hardware projectors, transforming an input dataset into a higher dimensional speckled image set. The goal of our study is to demonstrate that using such randomized data for classification by training a single logistic regression layer improves accuracy compared to training on direct raw images. Interestingly, we found that the classification accuracy achieved is higher than that obtained with the standard transmission matrix model, a widely accepted tool for describing light transmission through disordered devices. We conjecture that the reason for such improved performance could be due to the fact that the hardware classifier operates in a flatter region of the loss landscape when trained on fiber data, which aligns with the current theory of deep neural networks. These findings suggest that the class of random projections operated by multi-mode fibers generalize better to previously unseen data, positioning them as promising tools for optically-assisted neural networks. With this study, in fact, we want to contribute to advancing the knowledge and practical utilization of these versatile instruments, which may play a significant role in shaping the future of neuromorphic machine learning.

physics.optics↗

arXiv2025

SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation

Spurred by the demand for interpretable models, research on eXplainable AI for language technologies has experienced significant growth, with feature attribution methods emerging as a cornerstone of this progress. While prior work in NLP explored such methods for classification tasks and textual applications, explainability intersecting generation and speech is lagging, with existing techniques failing to account for the autoregressive nature of state-of-the-art models and to provide fine-grained, phonetically meaningful explanations. We address this gap by introducing Spectrogram Perturbation for Explainable Speech-to-text Generation (SPES), a feature attribution technique applicable to sequence generation tasks with autoregressive models. SPES provides explanations for each predicted token based on both the input spectrogram and the previously generated tokens. Extensive evaluation on speech recognition and translation demonstrates that SPES generates explanations that are faithful and plausible to humans.

cs.CL↗

arXiv2025

Energy release and Griffith's criterion for phase-field fracture

Phase field evolutions are obtained by means of time discrete schemes, providing (or selecting) at each time step an equilibrium configuration of the system, which is usually computed by descent methods for the free energy (e.g.staggered and monolithic schemes) under a suitable irreversibility constraint on the phase-field parameter. We study in detail the time continuous limits of these evolutions considering monotonicity as irreversibility constraint and providing a general result, which holds independently of the scheme employed in the incremental problem. In particular, we show that in the steady state regime the limit evolution is simultaneous (in displacement and phase field parameter) and satisfies Griffith's criterion in terms of toughness and phase field energy release rate. In the unsteady regime the limit evolution may instead depend on the adopted scheme and Griffith's criterion may not hold. We prove also the thermodynamical consistency of the monotonicity constraint over the whole evolution, and we study the system of PDEs (actually, a weak variational inequality) in the steady state regime. Technically, the proof employs a suitable reparametrization of the time discrete points, whose Kuratowski limit characterizes the set of steady state propagation. The study of the quasi-static time continuous limit relies on the strong convergence of the phase field function together with the convergence of the power identity.

math.AP↗

arXiv2025

AT1 fourth-order isogeometric phase-field modeling of brittle fracture

A crucial aspect in phase-field modeling, based on the variational formulation of brittle fracture, is the accurate representation of how the fracture surface energy is dissipated during the fracture process in the energy competition within a minimization problem. In general, the family of AT1 functionals showcases a well-defined elastic limit and narrow transition regions before crack onset, as opposed to AT2 models. On the other hand, high-order functionals provide similar accuracy as low-order ones but allow for larger mesh sizes in their discretization, remarkably reducing the computational cost. In this work, we aim to combine both these advantages and propose a novel AT1 fourth-order phase-field model for brittle fracture within an isogeometric framework, which provides a straightforward discretization of the high-order term in the crack surface density functional. For the introduced AT1 functional, we first prove a Γ-convergence result (in both the continuum and discretized isogeometric setting) based on a careful study of the optimal transition profile, which ultimately provides the explicit correction factor for the toughness and the exact size of the transition region. Fracture irreversibility is modeled by monotonicity of the damage variable and is conveniently enforced using the Projected Successive Over-Relaxation algorithm. Our numerical results indicate that the proposed fourth-order AT1 model is more accurate than the considered lower-order AT1 and AT2 models; this allows to employ larger mesh sizes, entailing a lower computational cost.

math.NA↗

arXiv2024

GFG -- Gender-Fair Generation: A CALAMITA Challenge

Gender-fair language aims at promoting gender equality by using terms and expressions that include all identities and avoid reinforcing gender stereotypes. Implementing gender-fair strategies is particularly challenging in heavily gender-marked languages, such as Italian. To address this, the Gender-Fair Generation challenge intends to help shift toward gender-fair language in written communication. The challenge, designed to assess and monitor the recognition and generation of gender-fair language in both mono- and cross-lingual scenarios, includes three tasks: (1) the detection of gendered expressions in Italian sentences, (2) the reformulation of gendered expressions into gender-fair alternatives, and (3) the generation of gender-fair language in automatic translation from English to Italian. The challenge relies on three different annotated datasets: the GFL-it corpus, which contains Italian texts extracted from administrative documents provided by the University of Brescia; GeNTE, a bilingual test set for gender-neutral rewriting and translation built upon a subset of the Europarl dataset; and Neo-GATE, a bilingual test set designed to assess the use of non-binary neomorphemes in Italian for both fair formulation and translation tasks. Finally, each task is evaluated with specific metrics: average of F1-score obtained by means of BERTScore computed on each entry of the datasets for task 1, an accuracy measured with a gender-neutral classifier, and a coverage-weighted accuracy for tasks 2 and 3.

cs.CL↗

arXiv2024

Speech Foundation Models and Crowdsourcing for Efficient, High-Quality Data Collection

While crowdsourcing is an established solution for facilitating and scaling the collection of speech data, the involvement of non-experts necessitates protocols to ensure final data quality. To reduce the costs of these essential controls, this paper investigates the use of Speech Foundation Models (SFMs) to automate the validation process, examining for the first time the cost/quality trade-off in data acquisition. Experiments conducted on French, German, and Korean data demonstrate that SFM-based validation has the potential to reduce reliance on human validation, resulting in an estimated cost saving of over 40.0% without degrading final data quality. These findings open new opportunities for more efficient, cost-effective, and scalable speech data acquisition.

cs.CL↗

arXiv2024

Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?

The field of natural language processing (NLP) has recently witnessed a transformative shift with the emergence of foundation models, particularly Large Language Models (LLMs) that have revolutionized text-based NLP. This paradigm has extended to other modalities, including speech, where researchers are actively exploring the combination of Speech Foundation Models (SFMs) and LLMs into single, unified models capable of addressing multimodal tasks. Among such tasks, this paper focuses on speech-to-text translation (ST). By examining the published papers on the topic, we propose a unified view of the architectural solutions and training strategies presented so far, highlighting similarities and differences among them. Based on this examination, we not only organize the lessons learned but also show how diverse settings and evaluation approaches hinder the identification of the best-performing solution for each architectural building block and training choice. Lastly, we outline recommendations for future works on the topic aimed at better understanding the strengths and weaknesses of the SFM+LLM solutions for ST.

cs.CL↗

arXiv2024

Findings of the IWSLT 2024 Evaluation Campaign

This paper reports on the shared tasks organized by the 21st IWSLT Conference. The shared tasks address 7 scientific challenges in spoken language translation: simultaneous and offline translation, automatic subtitling and dubbing, speech-to-speech translation, dialect and low-resource speech translation, and Indic languages. The shared tasks attracted 18 teams whose submissions are documented in 26 system papers. The growing interest towards spoken language translation is also witnessed by the constantly increasing number of shared task organizers and contributors to the overview paper, almost evenly distributed across industry and academia.

cs.CL↗