arXiv subjects

Ivan Titov

Publications and source records attributed to Ivan Titov.

At least 55 records · Page 3Linked to original sources

arXiv2024

Solovay reducibility implies S2a-reducibility

The original notion of Solovay reducibility was introduced by Robert M. Solovay (unpublished notes) in 1975 as a measure of relative randomness. The S2a-reducibility introduced by Xizhong Zheng and Robert Rettinger (DOI:10.1007/978-3-540-27798-9_39) in 2004 is a modification of Solovay reducibility suitable for computably approximable (c.a.) reals. We demonstrate that Solovay reducibility implies S2a-reducibility on the set of c.a. reals, even with the same constant, but not vice versa.

math.LO↗

arXiv2024

A total Solovay reducibility and totalizing of the notion of speedability

While the set of Martin-Löf random left-c.e. reals is equal to the maximum degree of Solovay reducibility, Miyabe, Nies and Stephan(DOI:10.4115/jla.2018.10.3) have shown that the left-c.e. Schnorr random reals are not closed upwards under Solovay reducibility. Recall that for two left-c.e. reals~$α$ and~$β$, the former is Solovay reducible to the latter in case there is a partially computable function $φ$ and constant~$c$ such that for all rational numbers $q < β$ we have \[α- φ(q) < c(β- q).\] By requiring the translation function $φ$ to be total, we introduce a total version of Solovay reducibility that implies Schnorr reducibility. Accordingly, by Downey and Griffiths (DOI:10.2178/jsl/1082418542), the set of Schnorr random left-c.e. reals is closed upwards relative to total Solovay reducibility. Furthermore, we observe that the notion of speedability introduced by Merkle and Titov (DOI:10.1007/978-3-030-50026-9_22) can be equivalently characterized via partial computable translation functions in a way that resembles Solovay reducibility. By requiring the translation function to be total, we obtain the concept of total speedability. Like for speedability, this notion does not depend on the choice of the speeding constant.

math.LO↗

arXiv2024

Extending the Limit Theorem of Barmpalias and Lewis-Pye to all reals

By a celebrated result of Kučera and Slaman (DOI:10.1137/S0097539799357441), the Martin-Löf random left-c.e. reals form the highest left-c.e. Solovay degree. Barmpalias and Lewis-Pye (arXiv:1604.00216) strengthened this result by showing that, for all left-c.e. reals $α$ and $β$ such that $β$ is Martin-Löf random and all left-c.e. approximations $a_0,a_1,\dots$ and $b_0,b_1,\dots$ of $α$ and $β$, respectively, the limit \begin{equation*} \lim\limits_{n\to\infty}\frac{α- a_n}{β- b_n} \end{equation*} exists and does not depend on the choice of the left-c.e. approximations to $α$ and $β$. Here we give an equivalent formulation of the result of Barmpalias and Lewis-Pye in terms of nondecreasing translation functions and generalize their result to the set of all (i.e., not necessarily left-c.e.) reals.

math.LO↗

arXiv2024

SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation

Strong inductive biases enable learning from little data and help generalization outside of the training distribution. Popular neural architectures such as Transformers lack strong structural inductive biases for seq2seq NLP tasks on their own. Consequently, they struggle with systematic generalization beyond the training distribution, e.g. with extrapolating to longer inputs, even when pre-trained on large amounts of text. We show how a structural inductive bias can be efficiently injected into a seq2seq model by pre-training it to simulate structural transformations on synthetic data. Specifically, we inject an inductive bias towards Finite State Transducers (FSTs) into a Transformer by pre-training it to simulate FSTs given their descriptions. Our experiments show that our method imparts the desired inductive bias, resulting in improved systematic generalization and better few-shot learning for FST-like tasks. Our analysis shows that fine-tuned models accurately capture the state dynamics of the unseen underlying FSTs, suggesting that the simulation process is internalized by the fine-tuned model.

cs.CL↗

arXiv2024

Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations

Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are highly versatile and powerful, they can still benefit from enhanced structural inductive biases for seq2seq tasks, especially those involving syntactic transformations, such as converting active to passive voice or semantic parsing. In this paper, we propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform synthetically generated syntactic transformations of dependency trees given a description of the transformation. Our experiments confirm that this helps with few-shot learning of syntactic tasks such as chunking, and also improves structural generalization for semantic parsing. Our analysis shows that the intermediate pre-training leads to attention heads that keep track of which syntactic transformation needs to be applied to which token, and that the model can leverage these attention heads on downstream tasks.

cs.CL↗

arXiv2024

Unlearning Traces the Influential Training Data of Language Models

Identifying the training datasets that influence a language model's outputs is essential for minimizing the generation of harmful content and enhancing its performance. Ideally, we can measure the influence of each dataset by removing it from training; however, it is prohibitively expensive to retrain a model multiple times. This paper presents UnTrac: unlearning traces the influence of a training dataset on the model's performance. UnTrac is extremely simple; each training dataset is unlearned by gradient ascent, and we evaluate how much the model's predictions change after unlearning. Furthermore, we propose a more scalable approach, UnTrac-Inv, which unlearns a test dataset and evaluates the unlearned model on training datasets. UnTrac-Inv resembles UnTrac, while being efficient for massive training datasets. In the experiments, we examine if our methods can assess the influence of pretraining datasets on generating toxic, biased, and untruthful content. Our methods estimate their influence much more accurately than existing methods while requiring neither excessive memory space nor multiple checkpoints.

cs.CL↗

arXiv2024

Magnetic microstructure of nanocrystalline Fe-Nb-B alloys as seen by small-angle neutron and X-ray scattering

We have investigated the magnetic microstructure of two-phase Fe-Nb-B~based Nanoperm alloys using unpolarized small-angle neutron scattering (SANS) and small-angle X-ray scattering (SAXS). Our SANS analysis reveals a significantly large magnetic scattering contribution due to spin misalignment, primarily originating from the substantial jump in the longitudinal magnetization at the interfaces between the particles and the matrix. The magnetic scattering exhibits an angular anisotropy that resembles a clover-leaf-type pattern, consistent with the predictions of micromagnetic SANS theory. Analysis of the one-dimensional SANS data yields values for the exchange-stiffness constant and the average anisotropy and magnetostatic fields. The micromagnetic correlation lengths for all three samples exhibit a similar field variation with sizes ranging between about 10-35 nm. We also find that the nuclear and magnetic residual scattering component of the SANS cross section exhibits a similar $q$ dependency as the SAXS data. These findings further validate the applicability of the micromagnetic SANS theory, and the mesoscopic information obtained is important for the advancement of the soft magnetic properties of this class of material.

cond-mat.mtrl-sci↗

arXiv2024

Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection

Researchers and practitioners operating on a limited budget face the cost-performance trade-off dilemma. The challenging decision often centers on whether to use a large LLM with better performance or a smaller one with reduced costs. This has motivated recent research in the optimisation of LLM calls. Either a cascading strategy is used, where a smaller LLM or both are called sequentially, or a routing strategy is used, where only one model is ever called. Both scenarios are dependent on a decision criterion which is typically implemented by an extra neural model. In this work, we propose a simpler solution; we use only the uncertainty of the generations of the small LLM as the decision criterion. We compare our approach with both cascading and routing strategies using three different pairs of pre-trained small and large LLMs, on nine different tasks and against approaches that require an additional neural model. Our experiments reveal this simple solution optimally balances cost and performance, outperforming existing methods on 25 out of 27 experimental setups.

cs.CL↗

arXiv2024

Autoencoding Conditional Neural Processes for Representation Learning

Conditional neural processes (CNPs) are a flexible and efficient family of models that learn to learn a stochastic process from data. They have seen particular application in contextual image completion - observing pixel values at some locations to predict a distribution over values at other unobserved locations. However, the choice of pixels in learning CNPs is typically either random or derived from a simple statistical measure (e.g. pixel variance). Here, we turn the problem on its head and ask: which pixels would a CNP like to observe - do they facilitate fitting better CNPs, and do such pixels tell us something meaningful about the underlying image? To this end we develop the Partial Pixel Space Variational Autoencoder (PPS-VAE), an amortised variational framework that casts CNP context as latent variables learnt simultaneously with the CNP. We evaluate PPS-VAE over a number of tasks across different visual data, and find that not only can it facilitate better-fit CNPs, but also that the spatial arrangement and values meaningfully characterise image information - evaluated through the lens of classification on both within and out-of-data distributions. Our model additionally allows for dynamic adaption of context-set size and the ability to scale-up to larger images, providing a promising avenue to explore learning meaningful and effective visual representations.

cs.LG↗

arXiv2023

Cross-Modal Conceptualization in Bottleneck Models

Concept Bottleneck Models (CBMs) assume that training examples (e.g., x-ray images) are annotated with high-level concepts (e.g., types of abnormalities), and perform classification by first predicting the concepts, followed by predicting the label relying on these concepts. The main difficulty in using CBMs comes from having to choose concepts that are predictive of the label and then having to label training examples with these concepts. In our approach, we adopt a more moderate assumption and instead use text descriptions (e.g., radiology reports), accompanying the images in training, to guide the induction of concepts. Our cross-modal approach treats concepts as discrete latent variables and promotes concepts that (1) are predictive of the label, and (2) can be predicted reliably from both the image and text. Through experiments conducted on datasets ranging from synthetic datasets (e.g., synthetic images with generated descriptions) to realistic medical imaging datasets, we demonstrate that cross-modal learning encourages the induction of interpretable concepts while also facilitating disentanglement. Our results also suggest that this guidance leads to increased robustness by suppressing the reliance on shortcut features.

cs.CL↗

arXiv2023

Compositional Generalization for Data-to-Text Generation

Data-to-text generation involves transforming structured data, often represented as predicate-argument tuples, into coherent textual descriptions. Despite recent advances, systems still struggle when confronted with unseen combinations of predicates, producing unfaithful descriptions (e.g. hallucinations or omissions). We refer to this issue as compositional generalisation, and it encouraged us to create a benchmark for assessing the performance of different approaches on this specific problem. Furthermore, we propose a novel model that addresses compositional generalization by clustering predicates into groups. Our model generates text in a sentence-by-sentence manner, relying on one cluster of predicates at a time. This approach significantly outperforms T5~baselines across all evaluation metrics.Notably, it achieved a 31% improvement over T5 in terms of a metric focused on maintaining faithfulness to the input.

cs.CL↗

arXiv2023

Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Study

With the ever-growing presence of social media platforms comes the increased spread of harmful content and the need for robust hate speech detection systems. Such systems easily overfit to specific targets and keywords, and evaluating them without considering distribution shifts that might occur between train and test data overestimates their benefit. We challenge hate speech models via new train-test splits of existing datasets that rely on the clustering of models' hidden representations. We present two split variants (Subset-Sum-Split and Closest-Split) that, when applied to two datasets using four pretrained models, reveal how models catastrophically fail on blind spots in the latent space. This result generalises when developing a split with one model and evaluating it on another. Our analysis suggests that there is no clear surface-level property of the data split that correlates with the decreased performance, which underscores that task difficulty is not always humanly interpretable. We recommend incorporating latent feature-based splits in model development and release two splits via the GenBench benchmark.

cs.CL↗

arXiv2023

Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation

When training a neural network, it will quickly memorise some source-target mappings from your dataset but never learn some others. Yet, memorisation is not easily expressed as a binary feature that is good or bad: individual datapoints lie on a memorisation-generalisation continuum. What determines a datapoint's position on that spectrum, and how does that spectrum influence neural models' performance? We address these two questions for neural machine translation (NMT) models. We use the counterfactual memorisation metric to (1) build a resource that places 5M NMT datapoints on a memorisation-generalisation map, (2) illustrate how the datapoints' surface-level characteristics and a models' per-datum training signals are predictive of memorisation in NMT, (3) and describe the influence that subsets of that map have on NMT systems' performance.

cs.CL↗

arXiv2023

Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training

Representational spaces learned via language modeling are fundamental to Natural Language Processing (NLP), however there has been limited understanding regarding how and when during training various types of linguistic information emerge and interact. Leveraging a novel information theoretic probing suite, which enables direct comparisons of not just task performance, but their representational subspaces, we analyze nine tasks covering syntax, semantics and reasoning, across 2M pre-training steps and five seeds. We identify critical learning phases across tasks and time, during which subspaces emerge, share information, and later disentangle to specialize. Across these phases, syntactic knowledge is acquired rapidly after 0.5% of full training. Continued performance improvements primarily stem from the acquisition of open-domain knowledge, while semantics and reasoning tasks benefit from later boosts to long-range contextualization and higher specialization. Measuring cross-task similarity further reveals that linguistically related tasks share information throughout training, and do so more during the critical phase of learning than before or after. Our findings have implications for model interpretability, multi-task learning, and learning from limited data.

cs.CL↗

arXiv2023

An Empirical Study of Compound PCFGs

Compound probabilistic context-free grammars (C-PCFGs) have recently established a new state of the art for unsupervised phrase-structure grammar induction. However, due to the high space and time complexities of chart-based representation and inference, it is difficult to investigate C-PCFGs comprehensively. In this work, we rely on a fast implementation of C-PCFGs to conduct an evaluation complementary to that of~\citet{kim-etal-2019-compound}. We start by analyzing and ablating C-PCFGs on English treebanks. Our findings suggest that (1) C-PCFGs are data-efficient and can generalize to unseen sentence/constituent lengths; and (2) C-PCFGs make the best use of sentence-level information in generating preterminal rule probabilities. We further conduct a multilingual evaluation of C-PCFGs. The experimental results show that the best configurations of C-PCFGs, which are tuned on English, do not always generalize to morphology-rich languages.

cs.CL↗

arXiv2023

On the Transferability of Visually Grounded PCFGs

There has been a significant surge of interest in visually grounded grammar induction in recent times. While a variety of models have been developed for the task and have demonstrated impressive performance, they have not been evaluated on text domains that are different from the training domain, so it is unclear if the improvements brought by visual groundings are transferable. Our study aims to fill this gap and assess the degree of transferability. We start by extending VC-PCFG (short for Visually-grounded Compound PCFG~\citep{zhao-titov-2020-visually}) in such a way that it can transfer across text domains. We consider a zero-shot transfer learning setting where a model is trained on the source domain and is directly applied to target domains, without any further training. Our experimental results suggest that: the benefits from using visual groundings transfer to text in a domain similar to the training domain but fail to transfer to remote domains. Further, we conduct data and result analysis; we find that the lexicon overlap between the source domain and the target domain is the most important factor in the transferability of VC-PCFG.

cs.CL↗

arXiv2023

Cache & Distil: Optimising API Calls to Large Language Models

Large-scale deployment of generative AI tools often depends on costly API calls to a Large Language Model (LLM) to fulfil user queries. To curtail the frequency of these calls, one can employ a smaller language model -- a student -- which is continuously trained on the responses of the LLM. This student gradually gains proficiency in independently handling an increasing number of user requests, a process we term neural caching. The crucial element in neural caching is a policy that decides which requests should be processed by the student alone and which should be redirected to the LLM, subsequently aiding the student's learning. In this study, we focus on classification tasks, and we consider a range of classic active learning-based selection criteria as the policy. Our experiments suggest that Margin Sampling and Query by Committee bring consistent benefits across tasks and budgets.

cs.CL↗

arXiv2023

Fingerprint of vortex-like flux closure in isotropic Nd-Fe-B bulk magnet

Taking advantage of recent progress in neutron instrumentation and in the understanding of magnetic-field-dependent small-angle neutron scattering, here, we study the three-dimensional magnetization distribution within an isotropic Nd-Fe-B bulk magnet. The magnetic neutron scattering cross section of this system features the so-called spike anisotropy, which points towards the presence of a strong magnetodipolar interaction. This experimental result combined with a damped oscillatory behavior of the corresponding correlation function and recent micromagnetic simulation results on spherical nanoparticles suggest an interpretation of the neutron data in terms of vortex-like flux-closure patterns. The field-dependent correlation length Lc is well reproduced by a phenomenological power-law model. While the experimental neutron data for Lc are described by an exponent close to unity (p = 0.86), the simulation results yield p = 1.70, posing a challenge to theory to include vortex-vortex interaction effects.

cond-mat.mes-hall↗