arXiv subjects

Igor Halperin

Publications and source records attributed to Igor Halperin.

At least 19 recordsLinked to original sources

arXiv2026

Are Three Matrices All You Need To Beat the Market? Observable Matrix Dynamics for Portfolio Optimization

We present a simple framework for dynamic portfolio management that uses nothing but daily prices, trading volumes, and market capitalizations. Its state is three fixed-size matrices built from the price history: the distance matrix of the return correlations and the transition matrices of two Markov chains that rank the S\&P 500 names monthly by trailing return and by trailing volatility. These three matrices rest on the price history alone, the same information Markowitz mean-variance optimization draws on, but they replace its expected-return vector and covariance matrix. Our method requires no matrix inversion, works on outlier-robust cross-sectional ranks, and is dynamic rather than single-period. Empirically the volatility rank is forecastable one step ahead while the return rank stays close to unforecastable. A portfolio built on the forecasts, a market-neutral momentum long-short blended with an opportunistic long-only sleeve, beats the market on two non-overlapping out-of-sample test sets, January 2022 to December 2024 and January 2025 to July 2026, at Sharpes of $1.06$ and $1.32$ against the market's $0.78$ and $1.14$, respectively, net of a five-basis-point trading cost and marked to market daily. It also outperforms the classical minimum-variance and maximum-diversification portfolios. Diversifying the long sleeve by residual distance adds a further edge on both periods, lifting the Sharpe to $1.08$ and $1.44$ and the annualized return from $18\%$ to $20\%$ and from $44\%$ to $56\%$, respectively. A convex information-leader overlay separately insures the market-neutral sleeve, buying convexity and a shallower drawdown at a small cost in return, the Sharpe unchanged.

q-fin.PM↗

arXiv2026

Observable Matrix Dynamics of Stocks

The Observable Matrix Dynamics (OMD) approach monitors the time development of complex non-linear systems through the trajectory of a fixed-size distance matrix and its spectrum. We apply it to the S\&P 500 cross section over three crisis decades, the 2001 dot-com bust, the 2007--2008 financial crisis, and the 2020 Covid crash, with three fixed-size observables on a fixed universe. The arccos distance matrix of the rolling return correlations reads the correlation geometry: its effective dimension collapses at the 2008 and 2020 crises, while the 2001 bust is a dispersed unwind. Read against machine-learning distance matrices, its spectrum stays in the un-relaxed, pre-learning regime with no low-dimensional manifold, so the market never learns its correlation structure or relaxes to a stationary geometry. Subtracting the market factor exposes a coherent sector rotation, whose name-level attribution identifies which stocks drive each crisis and in what order. At a short lookback these signals resolve precursors and forecast the endogenous 2008 crisis, though not the exogenous 2020 shock. The other two observables model the daily return and volatility rankings as Markov chains on their ranking spaces. The return chain has persistent, defensive-led bellwethers and near-reversible dynamics. The volatility chain is far more persistent, led by the financial sector, and is the only one to carry a weak, episodic arrow of time, flaring at market stress and matching volatility clustering and the Zumbach effect. All three matrices show coherent changes during market crashes.

q-fin.ST↗

arXiv2026

SciPhy Reinforcement Learning for Portfolio Optimization

This paper introduces a dynamic portfolio optimization framework for large institutional investors using Scientific Physics-Informed Reinforcement Learning (SciPhyRL). Formulated in continuous time over an extended state space that includes explicit cumulative costs, the approach leverages offline historical data to learn optimal, distribution-aware strategies. A core innovation reduces the optimization challenge to solving an HJB equation by projecting it onto observed trajectories as a pathwise Hamilton-Jacobi equation. This is solved directly from data using PINN in a single offline sweep, eliminating the need for traditional value or policy iteration. To make the method effective at practical short horizons, the control variable is recast from a continuous trading rate to a discrete target holding. This ensures signal-implied positions are reached immediately, while execution costs are evaluated against a microstructure-grounded quadratic price impact model. Evaluated on a $14$-asset ETF universe using an engineered oracle signal, the learned Gibbs policy yields substantial out-of-sample Sharpe ratio improvements over static and myopic baselines. The results demonstrate that the proposed framework successfully translates known signal quality into a robust, multi-period, and cost-aware allocation mechanism with strictly controlled volatility and turnover.

q-fin.PM↗

arXiv2026

I-BBS: Coordinate-Free Inference of Latent Sub-Manifolds Using Random Distance Matrix Theory

Bogomolny, Bohigas and Schmit (BBS) found that the spectrum of the pairwise distance matrix on N points sampled from a smooth d-dimensional manifold encodes a signature of the underlying geometry. We develop I-BBS (Inference-BBS), a coordinate-free method that identifies a low-dimensional latent sub-manifold embedded in a high-dimensional ambient distance matrix alone, without accessing an ambient high-dimensional vector space. It therefore applies even when that space is only partly observable or undefined. We model the ambient embedding by two classes of generative noise, model-based and model-free. The noise mixes the latent signal with off-manifold components, so the eigenvalues reorganise collectively and the latent geometry cannot be read off eigenvalue by eigenvalue. We recover it instead from two integer-stable signatures that survive the noise: the multiplicity of the top non-Perron multiplet, which fixes $d$, and a parameter-free law for how the multiplet positions shrink as the noise grows. On synthetic spheres $S^1$, $S^2$ and $S^3$ these integer signatures are far more stable under noise than the continuous spectral slope, and a blind test recovers both the manifold and the noise model from a single distance matrix. Applications to neural-network representations and to the dynamic training regime are developed in two companion papers.

cs.LG↗

arXiv2026

Learning as Observable Matrix Dynamics: Diffusive Relaxations versus Phase Transitions

Observable Matrix Dynamics (OMD) is a diagnostic framework that probes the dynamics of high-dimensional internal representations of inputs by a neural network via a fixed-size $N \times N$ distance matrix $M(t)$ on a held set of $N$ inputs. OMD uses methods of random matrix theory and particle dynamics to explore spectral reorganisations that are missed by scalar loss functions, but are informative of the training process. We read $M(t)$ against a perturbative ambient-versus-latent decomposition extending the Bogomolny--Bohigas--Schmit (BBS) theory of random distance matrices, with per-snapshot diagnostics for the top-of-spectrum band structure and ambient noise, trajectory-level observables linking snapshots, and a 3D MDS embedding (bottom-three eigenvectors) rendering training as a moving particle cloud. Across seven experiments, diffusive regimes lack stable top-of-spectrum band structure, while sharp endogenous or externally driven reorganisations produce stable fingerprints: consistent with smooth or product latent geometries in BBS-adjacent cases, and with finite-cluster or Fourier-soliton structures otherwise. OMD thus reads the geometric regime of a representation rather than reporting a single intrinsic dimension.

cs.LG↗

arXiv2026

Frustrated Fields: Statistical Field Theory for Frustrated Brownian Particles on 2D Manifolds

We develop a statistical field theory that describes the large-N limit of a system of Brownian particles with quenched random pairwise interactions on a compact two-dimensional Riemannian manifold. The resulting Frustrated Fields (F2) model is a non-linear field theory for a smooth self-interacting density field $\rho$ on the manifold, with local and non-local (in space and time) self-interactions characteristic of spin-glass dynamics. Particle simulations show \emph{adiabatic dimension reduction}: on $S^2$, the density concentrates on a slowly precessing great-circle ring whose orientation is a director ($\hat{\mathbf{n}} \sim -\hat{\mathbf{n}}$, even profile). Conditioned on this simulation-supported ring saddle and on a Born-Oppenheimer separation between the slow orientation and the gapped density fluctuations, symmetry fixes the low-energy dynamics to be the nonlinear sigma model (NLSM) on the real projective plane $S^2/\mathbb{Z}_2 = \mathbb{RP}^2$ (the $\mathbb{RP}^2$ NLSM on the projective rotor space) in $(0+1)$ dimensions, governed by a single low-energy constant, the rotational diffusion coefficient $D_{\text{rot}}$. With $D_{\text{rot}}$ and the static ring profile $f_0$ measured from particle simulations, the resulting effective theory reproduces multiple independent orientation- and density-sector diagnostics with no further adjustable parameters.

cond-mat.stat-mech↗

arXiv2026

Frustrated Dynamics of Distance Matrices

We introduce the Frustrated Distance Matrix (FDM) model, a dynamic extension of the static distance-matrix ensemble on S^2 analyzed by Bogomolny, Bohigas, and Schmit (BBS). Its entries are pairwise geodesic distances between N Brownian particles on the sphere evolving under quenched random pairwise couplings linear in those distances. Where the static BBS theory recovers geometric information about the underlying manifold from spectra of distance matrices on i.i.d.\ samples, the time-resolved FDM spectrum carries information about structural changes of the underlying point process. The particle dynamics realize one such change: a fast collapse from a uniform configuration onto a one-dimensional ring, followed by slow rotational drift of the ring orientation; the particle-level picture provides the ground truth against which spectral diagnostics are calibrated. We find that the static BBS template is preserved at every time, with the dynamics entering as a redistribution of spectral mass within that template, sharp enough to flag ring formation. We propose self-averaging of the bulk density as the mechanism behind this preservation, verified by an i.i.d.-resample comparison, and extract a small set of spectral diagnostics of the structural change computable from the distance matrix alone. We suggest that our diagnostics can be applied in other similar inverse-problem settings: financial correlation matrices, graph and network adjacency spectra, similarity matrices in molecular dynamics, and dynamics on parameter manifolds.

nlin.PS↗

arXiv2026

Order Out of Noise and Disorder: Fate of the Frustrated Manifold

We study Langevin dynamics of $N$ Brownian particles on two-dimensional Riemannian manifolds, interacting through pairwise potentials linear in geodesic distance with quenched random couplings. These \emph{frustrated Brownian particles} experience competing demands of random attractive and repulsive interactions while confined to curved surfaces. We consider three geometries: the sphere $S^2$, torus $T^2$, and bounded cylinder. Our central finding is disorder-induced dimension reduction with spontaneous rotational symmetry breaking: order emerges from two sources of randomness (thermal noise and quenched disorder), with manifold topology determining the character of emerging structures. Glassy relaxation drives particles from 2D distributions to quasi-1D structures: bands on $S^2$, rings on $T^2$, and localized clusters on the cylinder. Unlike conventional symmetry breaking, the symmetry-breaking direction is not frozen but evolves slowly via thermal noise. On the sphere, the structure normal precesses diffusively on the Goldstone manifold with correlation time $\tau_c \approx 18$, a classical realization of type-A dissipative Nambu-Goldstone dynamics. The model requires no thermodynamic gradients, no fine-tuning, and no slow external input. We discuss connections to spin glass theory, quantum field theory, astrophysical structure formation, and self-organizing systems. The model admits a large-$N$ limit yielding statistical field theory on Riemannian surfaces, while remaining experimentally realizable in colloidal and soft matter systems.

nlin.AO↗

arXiv2025

Physicists Are Still Joking

This volume, \textbf{Physicists Are Still Joking}, serves as a definitive almanac of scientific humor spanning sixty years. It traces the evolution of professional folklore across geopolitical divides and technological eras. \textbf{Part I} restores the classic 1966 anthology \textbf{Physicists Joke}, which originally served as a window for Soviet scientists into the best traditions of Western scientific humor; it consists primarily of articles translated from English, here meticulously restored to their original wording. \textbf{Part II} presents the 1992 sequel, \textbf{Physicists Keep Joking}, which captures the shift toward an original, introspective Russian scientific folklore born during the end of the Cold War and the collapse of the Soviet Union. \textbf{Part III: Still Joking} explores the modern digital age, compiling contemporary science humor from physics, astronomy, biology, computer science and AI research. While the tools of science have evolved from slide rules to neural networks, the tradition of skeptical, self-referential wit remains a constant. Spanning from the "Golden Age" of vacuum tubes to the era of AI and Large Language Models, this collection documents the enduring ability of scientists to laugh at the universe and themselves.

physics.soc-ph↗

arXiv2025

Semantic Faithfulness and Entropy Production Measures to Tame Your LLM Demons and Manage Hallucinations

Evaluating faithfulness of Large Language Models (LLMs) to a given task is a complex challenge. We propose two new unsupervised metrics for faithfulness evaluation using insights from information theory and thermodynamics. Our approach treats an LLM as a bipartite information engine where hidden layers act as a Maxwell demon controlling transformations of context $C $ into answer $A$ via prompt $Q$. We model Question-Context-Answer (QCA) triplets as probability distributions over shared topics. Topic transformations from $C$ to $Q$ and $A$ are modeled as transition matrices ${\bf Q}$ and ${\bf A}$ encoding the query goal and actual result, respectively. Our semantic faithfulness (SF) metric quantifies faithfulness for any given QCA triplet by the Kullback-Leibler (KL) divergence between these matrices. Both matrices are inferred simultaneously via convex optimization of this KL divergence, and the final SF metric is obtained by mapping the minimal divergence onto the unit interval [0,1], where higher scores indicate greater faithfulness. Furthermore, we propose a thermodynamics-based semantic entropy production (SEP) metric in answer generation, and show that high faithfulness generally implies low entropy production. The SF and SEP metrics can be used jointly or separately for LLM evaluation and hallucination control. We demonstrate our framework on LLM summarization of corporate SEC 10-K filings.

cs.AI↗

arXiv2025

From Text to Alpha: Can LLMs Track Evolving Signals in Corporate Disclosures?

Natural language processing (NLP) has been widely used in quantitative finance, but traditional methods often struggle to capture rich narratives in corporate disclosures, leaving potentially informative signals under-explored. Large language models (LLMs) offer a promising alternative due to their ability to extract nuanced semantics. In this paper, we ask whether semantic signals extracted by LLMs from corporate disclosures predict alpha, defined as abnormal returns beyond broad market movements and common risk factors. We introduce a simple framework, LLM as extractor, embedding as ruler, which extracts context-aware, metric-focused textual spans and quantifies semantic changes across consecutive disclosure periods using embedding-based similarity. This allows us to measure the degree of metric shifting -- how much firms move away from previously emphasized metrics, referred as moving targets. In experiments with portfolio and cross-sectional regression tests against a recent NER-based baseline, our method achieves more than twice the risk-adjusted alpha and shows significantly stronger predictive power. Qualitative analysis suggests that these gains stem from preserving contextual qualifiers and filtering out non-metric terms that keyword-based approaches often miss.

cs.CE↗

arXiv2025

Non-Linear and Meta-Stable Dynamics in Financial Markets: Evidence from High Frequency Crypto Currency Market Makers

This work builds upon the long-standing conjecture that linear diffusion models are inadequate for complex market dynamics. Specifically, it provides experimental validation for the author's prior arguments that realistic market dynamics are governed by higher-order (cubic and higher) non-linearities in the drift. As the diffusion drift is given by the negative gradient of a potential function, this means that a non-linear drift translates into a non-quadratic potential. These arguments were based both on general theoretical grounds as well as a structured approach to modeling the price dynamics which incorporates money flows and their impact on market prices. Here, we find direct confirmation of this view by analyzing high-frequency crypto currency data at different time scales ranging from minutes to months. We find that markets can be characterized by either a single-well or a double-well potential, depending on the time period and sampling frequency, where a double-well potential may signal market uncertainty or stress.

q-fin.ST↗

arXiv2025

Topic Identification in LLM Input-Output Pairs through the Lens of Information Bottleneck

Large Language Models (LLMs) are prone to critical failure modes, including \textit{intrinsic faithfulness hallucinations} (also known as confabulations), where a response deviates semantically from the provided context. Frameworks designed to detect this, such as Semantic Divergence Metrics (SDM), rely on identifying latent topics shared between prompts and responses, typically by applying geometric clustering to their sentence embeddings. This creates a disconnect, as the topics are optimized for spatial proximity, not for the downstream information-theoretic analysis. In this paper, we bridge this gap by developing a principled topic identification method grounded in the Deterministic Information Bottleneck (DIB) for geometric clustering. Our key contribution is to transform the DIB method into a practical algorithm for high-dimensional data by substituting its intractable KL divergence term with a computationally efficient upper bound. The resulting method, which we dub UDIB, can be interpreted as an entropy-regularized and robustified version of K-means that inherently favors a parsimonious number of informative clusters. By applying UDIB to the joint clustering of LLM prompt and response embeddings, we generate a shared topic representation that is not merely spatially coherent but is fundamentally structured to be maximally informative about the prompt-response relationship. This provides a superior foundation for the SDM framework and offers a novel, more sensitive tool for detecting confabulations.

cs.CL↗

arXiv2025

Marketron Through the Looking Glass: From Equity Dynamics to Option Pricing in Incomplete Markets

The Marketron model, introduced by [Halperin, Itkin, 2025], describes price formation in inelastic markets as the nonlinear diffusion of a quasiparticle (the marketron) in a multidimensional space comprising the log-price $x$, a memory variable $y$ encoding past money flows, and unobservable return predictors $z$. While the original work calibrated the model to S\&P 500 time series data, this paper extends the framework to option markets - a fundamentally distinct challenge due to market incompleteness stemming from non-tradable state variables. We develop a utility-based pricing approach that constructs a risk-adjusted measure via the dual solution of an optimal investment problem. The resulting Hamilton-Jacobi-Bellman (HJB) equation, though computationally formidable, is solved using a novel methodology enabling efficient calibration even on standard laptop hardware. Having done that, we look at the additional question to answer: whether the Marketron model, calibrated to market option prices, can simultaneously reproduce the statistical properties of the underlying asset's log-returns. We discuss our results in view of the long-standing challenge in quantitative finance of developing an unified framework capable of jointly capturing equity returns, option smile dynamics, and potentially volatility index behavior.

q-fin.PR↗

arXiv2025

Prompt-Response Semantic Divergence Metrics for Faithfulness Hallucination and Misalignment Detection in Large Language Models

The proliferation of Large Language Models (LLMs) is challenged by hallucinations, critical failure modes where models generate non-factual, nonsensical or unfaithful text. This paper introduces Semantic Divergence Metrics (SDM), a novel lightweight framework for detecting Faithfulness Hallucinations -- events of severe deviations of LLMs responses from input contexts. We focus on a specific implementation of these LLM errors, {confabulations, defined as responses that are arbitrary and semantically misaligned with the user's query. Existing methods like Semantic Entropy test for arbitrariness by measuring the diversity of answers to a single, fixed prompt. Our SDM framework improves upon this by being more prompt-aware: we test for a deeper form of arbitrariness by measuring response consistency not only across multiple answers but also across multiple, semantically-equivalent paraphrases of the original prompt. Methodologically, our approach uses joint clustering on sentence embeddings to create a shared topic space for prompts and answers. A heatmap of topic co-occurances between prompts and responses can be viewed as a quantified two-dimensional visualization of the user-machine dialogue. We then compute a suite of information-theoretic metrics to measure the semantic divergence between prompts and responses. Our practical score, $\mathcal{S}_H$, combines the Jensen-Shannon divergence and Wasserstein distance to quantify this divergence, with a high score indicating a Faithfulness hallucination. Furthermore, we identify the KL divergence KL(Answer $||$ Prompt) as a powerful indicator of \textbf{Semantic Exploration}, a key signal for distinguishing different generative behaviors. These metrics are further combined into the Semantic Box, a diagnostic framework for classifying LLM response types, including the dangerous, confident confabulation.

cs.CL↗

arXiv2024

CAISSON: Concept-Augmented Inference Suite of Self-Organizing Neural Networks

We present CAISSON, a novel hierarchical approach to Retrieval-Augmented Generation (RAG) that transforms traditional single-vector search into a multi-view clustering framework. At its core, CAISSON leverages dual Self-Organizing Maps (SOMs) to create complementary organizational views of the document space, where each view captures different aspects of document relationships through specialized embeddings. The first view processes combined text and metadata embeddings, while the second operates on metadata enriched with concept embeddings, enabling a comprehensive multi-view analysis that captures both fine-grained semantic relationships and high-level conceptual patterns. This dual-view approach enables more nuanced document discovery by combining evidence from different organizational perspectives. To evaluate CAISSON, we develop SynFAQA, a framework for generating synthetic financial analyst notes and question-answer pairs that systematically tests different aspects of information retrieval capabilities. Drawing on HotPotQA's methodology for constructing multi-step reasoning questions, SynFAQA generates controlled test cases where each question is paired with the set of notes containing its ground-truth answer, progressing from simple single-entity queries to complex multi-hop retrieval tasks involving multiple entities and concepts. Our experimental results demonstrate substantial improvements over both basic and enhanced RAG implementations, particularly for complex multi-entity queries, while maintaining practical response times suitable for interactive applications.

cs.CL↗

arXiv2023

SCOP: Schrodinger Control Optimal Planning for Goal-Based Wealth Management

We consider the problem of optimization of contributions of a financial planner such as a working individual towards a financial goal such as retirement. The objective of the planner is to find an optimal and feasible schedule of periodic installments to an investment portfolio set up towards the goal. Because portfolio returns are random, the practical version of the problem amounts to finding an optimal contribution scheme such that the goal is satisfied at a given confidence level. This paper suggests a semi-analytical approach to a continuous-time version of this problem based on a controlled backward Kolmogorov equation (BKE) which describes the tail probability of the terminal wealth given a contribution policy. The controlled BKE is solved semi-analytically by reducing it to a controlled Schrodinger equation and solving the latter using an algebraic method. Numerically, our approach amounts to finding semi-analytical solutions simultaneously for all values of control parameters on a small grid, and then using the standard two-dimensional spline interpolation to simultaneously represent all satisficing solutions of the original plan optimization problem. Rather than being a point in the space of control variables, satisficing solutions form continuous contour lines (efficient frontiers) in this space.

math.OC↗

arXiv2023

Model-Free Market Risk Hedging Using Crowding Networks

Crowding is widely regarded as one of the most important risk factors in designing portfolio strategies. In this paper, we analyze stock crowding using network analysis of fund holdings, which is used to compute crowding scores for stocks. These scores are used to construct costless long-short portfolios, computed in a distribution-free (model-free) way and without using any numerical optimization, with desirable properties of hedge portfolios. More specifically, these long-short portfolios provide protection for both small and large market price fluctuations, due to their negative correlation with the market and positive convexity as a function of market returns. By adding our long-short portfolio to a baseline portfolio such as a traditional 60/40 portfolio, our method provides an alternative way to hedge portfolio risk including tail risk, which does not require costly option-based strategies or complex numerical optimization. The total cost of such hedging amounts to the total cost of rebalancing the hedge portfolio.

q-fin.PM↗