arXiv subjects

Ramsés J. Sánchez

Publications and source records attributed to Ramsés J. Sánchez.

10 recordsLinked to original sources

arXiv2026

In-Context Learning of Temporal Point Processes with Foundation Inference Models

Modeling event sequences of multiple event types with marked temporal point processes (MTPPs) provides a principled way to uncover governing dynamical rules and predict future events. Current neural network approaches to MTPP inference rely on training separate, specialized models for each target system. We pursue a radically different approach: drawing on amortized inference and in-context learning, we pretrain a deep neural network to infer, in-context, the conditional intensity functions of event histories from a context defined by sets of event sequences. Pretraining is performed on a large synthetic dataset of MTPPs sampled from a broad distribution of Hawkes processes. Once pretrained, our Foundation Inference Model for Point Processes (FIM-PP) can estimate MTPPs from real-world data without any additional training, or be rapidly finetuned to target systems. Experiments show that this amortized approach matches the performance of specialized models on next-event prediction across common benchmark datasets.

cs.LG↗

arXiv2026

Prior-Fitted Functional Flow: In-Context Generative Models for Pharmacokinetics

We introduce Prior-Fitted Functional Flows, a generative foundation model for pharmacokinetics that enables zero-shot population synthesis and individual forecasting without manual parameter tuning. We learn functional vector fields, explicitly conditioned on the sparse, irregular data of an entire study population. This enables the generation of coherent virtual cohorts as well as forecasting of partially observed patient trajectories with calibrated uncertainty. We construct a new open-access literature corpus to inform our priors, and demonstrate state-of-the-art predictive accuracy on extensive real-world datasets.

cs.LG↗

arXiv2026

Amortized In-Context Mixed Effect Transformer Models: A Zero-Shot Approach for Pharmacokinetics

Accurate dose-response forecasting under sparse sampling is central to precision pharmacotherapy. We present the Amortized In-Context Mixed-Effect Transformer (AICMET) model, a transformer-based latent-variable framework that unifies mechanistic compartmental priors with amortized in-context Bayesian inference. AICMET is pre-trained on hundreds of thousands of synthetic pharmacokinetic trajectories with Ornstein-Uhlenbeck priors over the parameters of compartment models, endowing the model with strong inductive biases and enabling zero-shot adaptation to new compounds. At inference time, the decoder conditions on the collective context of previously profiled trial participants, generating calibrated posterior predictions for newly enrolled patients after a few early drug concentration measurements. This capability collapses traditional model-development cycles from weeks to hours while preserving some degree of expert modelling. Experiments across public datasets show that AICMET attains state-of-the-art predictive accuracy and faithfully quantifies inter-patient variability -- outperforming both nonlinear mixed-effects baselines and recent neural ODE variants. Our results highlight the feasibility of transformer-based, population-aware neural architectures as offering a new alternative for bespoke pharmacokinetic modeling pipelines, charting a path toward truly population-aware personalized dosing regimens.

cs.LG↗

arXiv2025

Zero-shot Imputation with Foundation Inference Models for Dynamical Systems

Dynamical systems governed by ordinary differential equations (ODEs) serve as models for a vast number of natural and social phenomena. In this work, we offer a fresh perspective on the classical problem of imputing missing time series data, whose underlying dynamics are assumed to be determined by ODEs. Specifically, we revisit ideas from amortized inference and neural operators, and propose a novel supervised learning framework for zero-shot time series imputation, through parametric functions satisfying some (hidden) ODEs. Our proposal consists of two components. First, a broad probability distribution over the space of ODE solutions, observation times and noise mechanisms, with which we generate a large, synthetic dataset of (hidden) ODE solutions, along with their noisy and sparse observations. Second, a neural recognition model that is trained offline, to map the generated time series onto the spaces of initial conditions and time derivatives of the (hidden) ODE solutions, which we then integrate to impute the missing data. We empirically demonstrate that one and the same (pretrained) recognition model can perform zero-shot imputation across 63 distinct time series with missing values, each sampled from widely different dynamical systems. Likewise, we demonstrate that it can perform zero-shot imputation of missing high-dimensional data in 10 vastly different settings, spanning human motion, air quality, traffic and electricity studies, as well as Navier-Stokes simulations -- without requiring any fine-tuning. What is more, our proposal often outperforms state-of-the-art methods, which are trained on the target datasets. Our pretrained model, repository and tutorials are available online.

cs.LG↗

arXiv2023

Hidden Schema Networks

Large, pretrained language models infer powerful representations that encode rich semantic and syntactic content, albeit implicitly. In this work we introduce a novel neural language model that enforces, via inductive biases, explicit relational structures which allow for compositionality onto the output representations of pretrained language models. Specifically, the model encodes sentences into sequences of symbols (composed representations), which correspond to the nodes visited by biased random walkers on a global latent graph, and infers the posterior distribution of the latter. We first demonstrate that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences. Next, we leverage pretrained BERT and GPT-2 language models as encoder and decoder, respectively, to infer networks of symbols (schemata) from natural language datasets. Our experiments show that (i) the inferred symbols can be interpreted as encoding different aspects of language, as e.g. topics or sentiments, and that (ii) GPT-like models can effectively be conditioned on symbolic representations. Finally, we explore training autoregressive, random walk ``reasoning" models on schema networks inferred from commonsense knowledge databases, and using the sampled paths to enhance the performance of pretrained language models on commonsense If-Then reasoning tasks.

cs.CL↗

arXiv2023

Neural Dynamic Focused Topic Model

Topic models and all their variants analyse text by learning meaningful representations through word co-occurrences. As pointed out by Williamson et al. (2010), such models implicitly assume that the probability of a topic to be active and its proportion within each document are positively correlated. This correlation can be strongly detrimental in the case of documents created over time, simply because recent documents are likely better described by new and hence rare topics. In this work we leverage recent advances in neural variational inference and present an alternative neural approach to the dynamic Focused Topic Model. Indeed, we develop a neural model for topic evolution which exploits sequences of Bernoulli random variables in order to track the appearances of topics, thereby decoupling their activities from their proportions. We evaluate our model on three different datasets (the UN general debates, the collection of NeurIPS papers, and the ACL Anthology dataset) and show that it (i) outperforms state-of-the-art topic models in generalization tasks and (ii) performs comparably to them on prediction tasks, while employing roughly the same number of parameters, and converging about two times faster. Source code to reproduce our experiments is available online.

cs.CL↗

arXiv2022

The future is different: Large pre-trained language models fail in prediction tasks

Large pre-trained language models (LPLM) have shown spectacular success when fine-tuned on downstream supervised tasks. Yet, it is known that their performance can drastically drop when there is a distribution shift between the data used during training and that used at inference time. In this paper we focus on data distributions that naturally change over time and introduce four new REDDIT datasets, namely the WALLSTREETBETS, ASKSCIENCE, THE DONALD, and POLITICS sub-reddits. First, we empirically demonstrate that LPLM can display average performance drops of about 88% (in the best case!) when predicting the popularity of future posts from sub-reddits whose topic distribution changes with time. We then introduce a simple methodology that leverages neural variational dynamic topic models and attention mechanisms to infer temporal language model representations for regression tasks. Our models display performance drops of only about 40% in the worst cases (2% in the best ones) when predicting the popularity of future posts, while using only about 7% of the total number of parameters of LPLM and providing interpretable representations that offer insight into real-world events, like the GameStop short squeeze of 2021

cs.CL↗

arXiv2019

Recurrent Adversarial Service Times

Service system dynamics occur at the interplay between customer behaviour and a service provider's response. This kind of dynamics can effectively be modeled within the framework of queuing theory where customers' arrivals are described by point process models. However, these approaches are limited by parametric assumptions as to, for example, inter-event time distributions. In this paper, we address these limitations and propose a novel, deep neural network solution to the queuing problem. Our solution combines a recurrent neural network that models the arrival process with a recurrent generative adversarial network which models the service time distribution. We evaluate our methodology on various empirical datasets ranging from internet services (Blockchain, GitHub, Stackoverflow) to mobility service systems (New York taxi cab).

stat.ML↗

arXiv2018

Anomalous and regular transport in spin 1/2 chains: AC conductivity

We study magnetization transport in anisotropic spin-$1/2$ chains governed by the integrable XXZ model with and without integrability-breaking perturbations at high temperatures ($T\to \infty$) using a hybrid approach that combines exact sum-rules with judiciously chosen Ansätze. In the integrable XXZ model we find (i) super-diffusion at the isotropic (Heisenberg) point, with frequency dependent conductivity $ σ'(ω\to 0) \sim |ω|^α$, where $α=-3/7$ in close numerical agreement with recent $t$-DMRG computations; (ii) a continuously drifting exponent from $α=-1^+$ in the XY limit of the model to $α>0$ within the Ising regime; and (iii) a diffusion constant saturating in the XY coupling deep in the Ising limit. We consider two kinds of integrability breaking perturbations --- a simple next-nearest-neighbor spin-flip term ($J_2$) and a three-spin assisted variant ($t_2$), natural in the fermion particle representation of the spin chain. In the first case we discover a remarkable sensitivity of $σ'(ω)$ to the sign of $J_2$, with enhanced low frequency spectral weight and a pronounced upward shift in the magnitude of $α$ for $J_2>0$. Perhaps even more surprising, we find sub-diffusion ($α>0$) over a range of $J_2<0$. By contrast, the effects of the \enquote{fermionic} three-spin perturbation are sign symmetric; this perturbation produces a clearly observable hydrodynamic relaxation. At large strength of the integrability breaking term $J_2\to \pm \infty$ the problem is effectively non-interacting (fermions hopping on odd and even sublattices) and we find $α\to -1$ behavior reminiscent of the XY limit of the XXZ chain. Exact diagonalization studies largely corroborate these findings at mid-frequencies.

cond-mat.str-el↗

arXiv2017

Finite-size anomalies of the Drude weight: role of symmetries and ensembles

We revisit the subtelties of computing the high temperature spin stiffness $D$ of the spin-$1/2$ XXZ chain using exact diagonalization to analyze its dependence on system symmetries and ensemble. Within the canonical ensemble and for states with zero magnetization, we find $D$ vanishes exactly due to spin-inversion symmetry for all but the anisotropies $\tilde Δ_{MN} = \cos(πM /N)$ with $N > M$ and coprime, provided system sizes $L \ge 2N$, for which states with different spin-inversion signature become degenerate due to the underlying $sl_2$ loop algebra symmetry. All these loop-algebra degenerate states carry finite currents which we conjecture [based on $L$ and anisotropies $\tilde Δ_{MN}$ (with $N<L/2$) available to us] to dominate the grand-canonical ensemble evaluation of $D$ in the thermodynamic limit. Including a magnetic flux not only breaks spin-inversion in the zero magnetization sector but also lifts the loop-algebra degeneracies in all symmetry sectors --- this effect is more pertinent at smaller $Δ$ due to the larger contributions to $D$ coming from the low-magnetization sectors which are more sensitive to the system's symmetries. Thus we generically find a finite $D$ for fluxed rings and arbitrary $0<Δ<1$ in both ensembles. In contrast, at the isotropic point and in the gapped phase ($Δ\ge 1$) $D$ is found to vanish in the thermodynamic limit, independent of symmetry or ensemble. Our analysis demonstrates how convergence to the thermodynamic limit within the gapless phase ($Δ< 1$) may be accelerated and the finite-size anomalies overcome: $D$ extrapolates nicely in the thermodynamic limit to either the recently computed lower-bound or the Thermodynamic Bethe Ansatz result provided both spin-inversion is broken and the additional degeneracies at the $\tilde Δ_{MN}$ anisotropies are lifted.

cond-mat.str-el↗