arXiv subjects

Lucas Möller

Publications and source records attributed to Lucas Möller.

6 recordsLinked to original sources

arXiv2025

Interpretable Text Embeddings and Text Similarity Explanation: A Survey

Text embeddings are a fundamental component in many NLP tasks, including classification, regression, clustering, and semantic search. However, despite their ubiquitous application, challenges persist in interpreting embeddings and explaining similarities between them. In this work, we provide a structured overview of methods specializing in inherently interpretable text embeddings and text similarity explanation, an underexplored research area. We characterize the main ideas, approaches, and trade-offs. We compare means of evaluation, discuss overarching lessons learned and finally identify opportunities and open challenges for future research.

cs.CL↗

arXiv2025

Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributions

Dual encoder architectures like Clip models map two types of inputs into a shared embedding space and predict similarities between them. Despite their wide application, it is, however, not understood how these models compare their two inputs. Common first-order feature-attribution methods explain importances of individual features and can, thus, only provide limited insights into dual encoders, whose predictions depend on interactions between features. In this paper, we first derive a second-order method enabling the attribution of predictions by any differentiable dual encoder onto feature-interactions between its inputs. Second, we apply our method to Clip models and show that they learn fine-grained correspondences between parts of captions and regions in images. They match objects across input modes and also account for mismatches. This intrinsic visual-linguistic grounding ability, however, varies heavily between object classes, exhibits pronounced out-of-domain effects and we can identify individual errors as well as systematic failure categories. Code is publicly available: https://github.com/lucasmllr/exCLIP

cs.CV↗

arXiv2024

Approximate Attributions for Off-the-Shelf Siamese Transformers

Siamese encoders such as sentence transformers are among the least understood deep models. Established attribution methods cannot tackle this model class since it compares two inputs rather than processing a single one. To address this gap, we have recently proposed an attribution method specifically for Siamese encoders (Möller et al., 2023). However, it requires models to be adjusted and fine-tuned and therefore cannot be directly applied to off-the-shelf models. In this work, we reassess these restrictions and propose (i) a model with exact attribution ability that retains the original model's predictive performance and (ii) a way to compute approximate attributions for off-the-shelf models. We extensively compare approximate and exact attributions and use them to analyze the models' attendance to different linguistic aspects. We gain insights into which syntactic roles Siamese transformers attend to, confirm that they mostly ignore negation, explore how they judge semantically opposite adjectives, and find that they exhibit lexical bias.

cs.CL↗

arXiv2023

An Attribution Method for Siamese Encoders

Despite the success of Siamese encoder models such as sentence transformers (ST), little is known about the aspects of inputs they pay attention to. A barrier is that their predictions cannot be attributed to individual features, as they compare two inputs rather than processing a single one. This paper derives a local attribution method for Siamese encoders by generalizing the principle of integrated gradients to models with multiple inputs. The solution takes the form of feature-pair attributions, and can be reduced to a token-token matrix for STs. Our method involves the introduction of integrated Jacobians and inherits the advantageous formal properties of integrated gradients: it accounts for the model's full computation graph and is guaranteed to converge to the actual prediction. A pilot study shows that in an ST few token-pairs can often explain large fractions of predictions, and it focuses on nouns and verbs. For accurate predictions, it however needs to attend to the majority of tokens and parts of speech.

cs.CL↗

arXiv2022

Understanding the Relation of User and News Representations in Content-Based Neural News Recommendation

A number of models for neural content-based news recommendation have been proposed. However, there is limited understanding of the relative importances of the three main components of such systems (news encoder, user encoder, and scoring function) and the trade-offs involved. In this paper, we assess the hypothesis that the most widely used means of matching user and candidate news representations is not expressive enough. We allow our system to model more complex relations between the two by assessing more expressive scoring functions. Across a wide range of baseline and established systems this results in consistent improvements of around 6 points in AUC. Our results also indicate a trade-off between the complexity of news encoder and scoring function: A fairly simple baseline model scores well above 68% AUC on the MIND dataset and comes within 2 points of the published state-of-the-art, while requiring a fraction of the computational costs.

cs.IR↗

arXiv2020

Filled Carbon Nanotubes as Anode Materials for Lithium-Ion Batteries

Downsizing well-established materials to the nanoscale is a key route to novel functionalities, in particular if different functionalities are merged in hybrid nanomaterials. Hybrid carbon-based hierarchical nanostructures are particularly promising for electrochemical energy storage since they combine benefits of nanosize effects, enhanced electrical conductivity and integrity of bulk materials. We show that endohedral multiwalled carbon nanotubes (CNT) encapsulating high-capacity (here: conversion and alloying) electrode materials have a high potential for use in anode materials for lithium-ion batteries (LIB). There are two essential characteristics of filled CNT relevant for application in electrochemical energy storage: (1) rigid hollow cavities of the CNT provide upper limits for nanoparticles in their inner cavities which are both separated from the fillings of other CNT and protected against degradation. In particular, the CNT shells resist strong volume changes of encapsulates in response to electrochemical cycling, which in conventional conversion and alloying materials hinders application in energy storage devices. (2) Carbon mantles ensure electrical contact to the active material as they are unaffected by potential cracks of the encapsulate and form a stable conductive network in the electrode compound. Our studies confirm that encapsulates are electrochemically active and can achieve full theoretical reversible capacity. The results imply that encapsulating nanostructures inside CNT can provide a route to new high-performance nanocomposite anode materials for LIB.

physics.app-ph↗