arXiv subjects

Thomas Cook

Publications and source records attributed to Thomas Cook.

8 recordsLinked to original sources

arXiv2025

Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQL

Large Language Models (LLMs) can generate SQL queries from natural language questions but struggle with database-specific schemas and tacit domain knowledge. We introduce a framework for continual learning from human feedback in text-to-SQL, where a learning agent receives natural language feedback to refine queries and distills the revealed knowledge for reuse on future tasks. This distilled knowledge is stored in a structured memory, enabling the agent to improve execution accuracy over time. We design and evaluate multiple variations of a learning agent architecture that vary in how they capture and retrieve past experiences. Experiments on the BIRD benchmark Dev set show that memory-augmented agents, particularly the Procedural Agent, achieve significant accuracy gains and error reduction by leveraging human-in-the-loop feedback. Our results highlight the importance of transforming tacit human expertise into reusable knowledge, paving the way for more adaptive, domain-aware text-to-SQL systems that continually learn from a human-in-the-loop.

cs.CL↗

arXiv2025

Retrieval Augmented Generation (RAG) for Fintech: Agentic Design and Evaluation

Retrieval-Augmented Generation (RAG) systems often face limitations in specialized domains such as fintech, where domain-specific ontologies, dense terminology, and acronyms complicate effective retrieval and synthesis. This paper introduces an agentic RAG architecture designed to address these challenges through a modular pipeline of specialized agents. The proposed system supports intelligent query reformulation, iterative sub-query decomposition guided by keyphrase extraction, contextual acronym resolution, and cross-encoder-based context re-ranking. We evaluate our approach against a standard RAG baseline using a curated dataset of 85 question--answer--reference triples derived from an enterprise fintech knowledge base. Experimental results demonstrate that the agentic RAG system outperforms the baseline in retrieval precision and relevance, albeit with increased latency. These findings suggest that structured, multi-agent methodologies offer a promising direction for enhancing retrieval robustness in complex, domain-specific settings.

cs.AI↗

arXiv2025

ScaleCall -- Agentic Tool Calling at Scale for Fintech: Challenges, Methods, and Deployment Insights

While Large Language Models (LLMs) excel at tool calling, deploying these capabilities in regulated enterprise environments such as fintech presents unique challenges due to on-premises constraints, regulatory compliance requirements, and the need to disambiguate large, functionally overlapping toolsets. In this paper, we present a comprehensive study of tool retrieval methods for enterprise environments through the development and deployment of ScaleCall, a prototype tool-calling framework within Mastercard designed for orchestrating internal APIs and automating data engineering workflows. We systematically evaluate embedding-based retrieval, prompt-based listwise ranking, and hybrid approaches, revealing that method effectiveness depends heavily on domain-specific factors rather than inherent algorithmic superiority. Through empirical investigation on enterprise-derived benchmarks, we find that embedding-based methods offer superior latency for large tool repositories, while listwise ranking provides better disambiguation for overlapping functionalities, with hybrid approaches showing promise in specific contexts. We integrate our findings into ScaleCall's flexible architecture and validate the framework through real-world deployment in Mastercard's regulated environment. Our work provides practical insights into the trade-offs between retrieval accuracy, computational efficiency, and operational requirements, contributing to the understanding of tool-calling system design for enterprise applications in regulated industries.

cs.SE↗

arXiv2024

Prune 'n Predict: Optimizing LLM Decision-making with Conformal Prediction

Large language models (LLMs) are empowering decision-making in several applications, including tool or API usage and answering multiple-choice questions (MCQs). However, incorrect outputs pose significant risks in high-stakes domains like healthcare and finance. To quantify LLM uncertainty and thereby mitigate these risks, recent works employ conformal prediction (CP), a model- and distribution-agnostic framework that uses LLM outputs to generate a \emph{prediction set} containing the true answer with high probability. Leveraging CP, we propose \emph{conformal revision of questions} (CROQ), which revises the question by narrowing down the available choices to those in the prediction set and asking the LLM the revised question. We expect LLMs to be more accurate on revised questions with fewer choices. Furthermore, we expect CROQ to be effective when the prediction sets from CP are small. Commonly used logit scores often lead to large sets, diminishing CROQ's effectiveness. To overcome this, we propose CP-OPT, an optimization framework to learn scores that minimize set sizes while maintaining coverage. Our extensive experiments on MMLU, ToolAlpaca, and TruthfulQA datasets with multiple LLMs show that CROQ improves accuracy over the standard inference, with more pronounced gains when paired with CP-OPT.

cs.LG↗

arXiv2024

Hedging in Sequential Experiments

Experimentation involves risk. The investigator expends time and money in the pursuit of data that supports a hypothesis. In the end, the investigator may find that all of these costs were for naught and the data fail to reject the null. Furthermore, the investigator may not be able to test other hypotheses with the same data set in order to avoid false positives due to p-hacking. Therefore, there is a need for a mechanism for investigators to hedge the risk of financial and statistical bankruptcy in the business of experimentation. In this work, we build on the game-theoretic statistics framework to enable an investigator to hedge their bets against the null hypothesis and thus avoid ruin. First, we describe a method by which the investigator's test martingale wealth process can be capitalized by solving for the risk-neutral price. Then, we show that a portfolio that comprises the risky test martingale and a risk-free process is still a test martingale which enables the investigator to select a particular risk-return position using Markowitz portfolio theory. Finally, we show that a function that is derivative of the test martingale process can be constructed and used as a hedging instrument by the investigator or as a speculative instrument by a risk-seeking investor who wants to participate in the potential returns of the uncertain experiment wealth process. Together, these instruments enable an investigator to hedge the risk of ruin and they enable a investigator to efficiently hedge experimental risk.

q-fin.RM↗

arXiv2023

Semiparametric Efficient Inference in Adaptive Experiments

We consider the problem of efficient inference of the Average Treatment Effect in a sequential experiment where the policy governing the assignment of subjects to treatment or control can change over time. We first provide a central limit theorem for the Adaptive Augmented Inverse-Probability Weighted estimator, which is semiparametric efficient, under weaker assumptions than those previously made in the literature. This central limit theorem enables efficient inference at fixed sample sizes. We then consider a sequential inference setting, deriving both asymptotic and nonasymptotic confidence sequences that are considerably tighter than previous methods. These anytime-valid methods enable inference under data-dependent stopping times (sample sizes). Additionally, we use propensity score truncation techniques from the recent off-policy estimation literature to reduce the finite sample variance of our estimator without affecting the asymptotic variance. Empirical results demonstrate that our methods yield narrower confidence sequences than those previously developed in the literature while maintaining time-uniform error control.

stat.ML↗

arXiv2022

Cost-aware Generalized $\alpha$-investing for Multiple Hypothesis Testing

We consider the problem of sequential multiple hypothesis testing with nontrivial data collection costs. This problem appears, for example, when conducting biological experiments to identify differentially expressed genes of a disease process. This work builds on the generalized $\alpha$-investing framework which enables control of the false discovery rate in a sequential testing setting. We make a theoretical analysis of the long term asymptotic behavior of $\alpha$-wealth which motivates a consideration of sample size in the $\alpha$-investing decision rule. Posing the testing process as a game with nature, we construct a decision rule that optimizes the expected $\alpha$-wealth reward (ERO) and provides an optimal sample size for each test. Empirical results show that a cost-aware ERO decision rule correctly rejects more false null hypotheses than other methods for $n=1$ where $n$ is the sample size. When the sample size is not fixed cost-aware ERO uses a prior on the null hypothesis to adaptively allocate of the sample budget to each test. We extend cost-aware ERO investing to finite-horizon testing which enables the decision rule to allocate samples in a non-myopic manner. Finally, empirical tests on real data sets from biological experiments show that cost-aware ERO balances the allocation of samples to an individual test against the allocation of samples across multiple tests.

cs.LG↗

arXiv2019

Heavy quarkonium suppression beyond the adiabatic limit

Many prior studies of in-medium quarkonium suppression have implicitly made use of an adiabatic approximation in which it was assumed that the heavy quark potential is a slowly varying function of time. In the adiabatic limit, one can separately determine the in-medium breakup rate and the medium time evolution, folding these together only at the end of the calculation. In this paper, we relax this assumption by solving the 3d Schrodinger equation in real-time in order to compute quarkonium suppression dynamically. We compare results obtained using the adiabatic approximation with real-time calculations for both harmonic oscillator and realistic complex heavy quark potentials. Using the latter, we find that, for the Upsilon(1s), the difference between the adiabatic approximation and full real-time evolution is at the few percent level, however, for the Upsilon(2s), we find that the correction can be as large as 18% in low temperature regions. For the J/Psi, we find a larger difference between the dynamical evolution and the adiabatic approximation, with the error reaching approximately 36%.

hep-ph↗