arXiv subjects

Thomas Benjamin Smith

Publications and source records attributed to Thomas Benjamin Smith.

7 recordsLinked to original sources

arXiv2025

Querying Databases with Function Calling

The capabilities of Large Language Models (LLMs) are rapidly accelerating largely thanks to their integration with external tools. Querying databases is among the most effective of these integrations, enabling LLMs to access private or continually updating data. While Function Calling is the most common method for interfacing external tools to LLMs, its application to database querying as a tool has been underexplored. We propose a tool definition for database querying that unifies accessing data with search queries, filters, or a combination both, as well as transforming results with aggregation and groupby operators. To evaluate its effectiveness, we conduct a study with 8 LLMs spanning 5 model families. We present a novel pipeline adapting the Gorilla LLM framework to create synthetic database schemas and queries. We primarily evaluate the models with the Exact Match of predicted and ground truth query APIs. Among the models tested, Claude 3.5 Sonnet achieves the highest performance with an Exact Match score of 74.3%, followed by GPT-4o mini at 73.7%, and GPT-4o at 71.8%. We further breakdown these results per API component utilized and across synthetic use cases. We find that LLMs are highly effective at utilizing operators on boolean properties, but struggle with text property filters. Across use cases we find robust results with the higher performing models such as GPT-4o, but significant performance variance across use cases from lower performing models. We additionally conduct ablation studies exploring the impact of parallel tool calling, adding a rationale as an argument of the tool call, using a separate tool per database collection, and tool calling with structured outputs. Our findings demonstrate the effectiveness of enabling LLMs to query databases with Function Calling. We have open-sourced our experimental code and results at github.com/weaviate/gorilla.

cs.DB↗

arXiv2024

StructuredRAG: JSON Response Formatting with Large Language Models

The ability of Large Language Models (LLMs) to generate structured outputs, such as JSON, is crucial for their use in Compound AI Systems. However, evaluating and improving this capability remains challenging. In this work, we introduce StructuredRAG, a benchmark of six tasks designed to assess LLMs' proficiency in following response format instructions. We evaluate two state-of-the-art LLMs, Gemini 1.5 Pro and Llama 3 8B-instruct with 4-bit quantization using two distinct prompting strategies. We introduce these prompting strategies as f-String and Follow the Format (FF) prompting. Across 24 experiments, we find an average success rate of 82.55%. We further find a high variance in performance across tasks, models, and prompting strategies with success rates ranging from 0 to 100%. We find that Llama 3 8B-instruct often performs competitively with Gemini 1.5 Pro. We observe that task complexity significantly influences performance, with tasks involving lists or composite object outputs proving more challenging. Our findings highlight the need for further research into improving the reliability and consistency of structured output generation in LLMs. We have open-sourced our experimental code and results at github.com/weaviate/structured-rag.

cs.CL↗

arXiv2020

Topological Surface-plasmon-polaritons on Corrugated Metal-dielectric Surfaces

We study topological surface-plasmon-polaritons at optical frequencies in diffraction gratings formed by bipartite corrugated metal-dielectric gratings. To do so we implement the theory as developed in Della Valle and Longhi to describe the amplitude of the field by an emergent Schrödinger-like equation. The tri-harmonic grating generates a bipartite Kronig-Penney model. Topologically protected localised modes are then predicted to occur at the edges of the grating and at defects formed by the combination of two mirror antisymmetric corrugations, whose bulk invariant is a step-wise varying Zak phase in both cases.

cond-mat.mes-hall↗

arXiv2020

Emergent Non-Hermitian Edge Polarisation in an Hermitian Tight-binding Model

We study a bipartite Kronig-Penney model with negative Dirac-delta potentials that may be used, amongst other models, to interpret plasmon propagation in nanoparticle arrays. Such a system can be mapped into a Su-Schrieffer-Heeger-like model however, in general, the overlap between 'atomic' wavefunctions of neighbouring sites is not negligible. In such a case, the edge states of the finite system, which retain their topological protection, appear to be either attenuated or amplified. This phenomenon, called "edge polarisation", is usually associated with an underlying non-Hermitian topology. By investigating the bulk system, we show that the resulting tight-binding eigenvalue problem may be made to appear non-Hermitian in this physical 'atomic' (lattice-site) basis. The resulting {\it effective} bulk Hamiltonian possesses ${\cal PT}$-symmetry and its topological invariant, interpreted in terms of a non-Hermitian classification, is found to be given by a bulk winding number of $\mathbb{Z}$-type. The observation of edge polarisation, through the established bulk-boundary correspondence, is then interpreted as an emerging non-Hermitian skin-effect of the {\it effective} bulk Hamiltonian. Therefore, the overlap matrix generates non-Hermitian-like effects in an otherwise Hermitian problem; a general fact applicable to a broader range of systems than just the one studied here.

cond-mat.mes-hall↗

arXiv2019

A Bipartite Kronig-Penney Model with Dirac Potential Scatterers

Here we present a simple extension to the age-old Kronig-Penney model, which is made to be bipartite by varying either the scatterer separations or the potential heights. In doing so, chiral (sublattice) symmetry can be introduced. When such a symmetry is present, topologically protected edge states are seen to exist. The solution proceeds through the conventional scattering formalism used to study the Kronig-Penney model, which does not require further tight-binding approximations or mapping into a Su-Schrieffer-Heeger model. The topological invariant for this specific system is found to be the winding of the reflection coefficient, ultimately linked to the system wavefunction. The solution of such a simple and illustrative 1D problem, whose topological content is extracted without requiring further tight-binding approximations, represents the novel aspect of our paper. The cases in which chiral symmetry is absent are then seen to not host topologically protected edge states, as verified by the behaviour of the reflection coefficient and the absence of winding.

cond-mat.mes-hall↗

arXiv2019

Edge modes and Fabry-Perot Plasmonic Resonances in anomalous-Hall Thin Films

We study plasmon propagation on a metallic two-dimensional surface partially coated with a thin film of anomalous-Hall material. The resulting three regions, separated by two sharp interfaces, are characterised by different Hall conductivities but identical normal conductivities. A single bound mode is found, which can localise to either interface and has an asymmetric potential profile across the region. For propagating modes, we calculate the reflection and transmission coefficients through the magnetic region. We find Airy transmission patterns with sharp maxima and minima as a function of the plasmon incidence angle. The system therefore behaves as a high-quality filter.

cond-mat.mes-hall↗

arXiv2019

Breakdown of the Wiedemann-Franz law in AB-stacked bilayer graphene

We present a simple theory of thermoelectric transport in bilayer graphene and report our results for the electrical resistivity, the thermal resistivity, the Seebeck coefficient, and the Wiedemann-Franz ratio as functions of doping density and temperature. In the absence of disorder, the thermal resistivity tends to zero as the charge neutrality point is approached; the electric resistivity jumps from zero to an intrinsic finite value, and the Seebeck coefficient diverges in the same limit. Even though these results are similar to those obtained for single-layer graphene, their derivation is considerably more delicate. The singularities are removed by the inclusion of a small amount of disorder, which leads to the appearance of a "window" of doping densities $0<n<n_c$ (with $n_c$ tending to zero in the zero-disorder limit) in which the Wiedemann-Franz law is severely violated.

cond-mat.mes-hall↗