Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey

Yoshiharu Sato

Home/Archive/arXiv

arXiv · 1904.04973

Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey

Yoshiharu Sato

View original source ↗Read source document ↗Export metadata ↓

Abstract

Financial portfolio management is one of the problems that are most frequently encountered in the investment industry. Nevertheless, it is not widely recognized that both Kelly Criterion and Risk Parity collapse into Mean Variance under some conditions, which implies that a universal solution to the portfolio optimization problem could potentially exist. In fact, the process of sequential computation of optimal component weights that maximize the portfolio's expected return subject to a certain risk budget can be reformulated as a discrete-time Markov Decision Process (MDP) and hence as a stochastic optimal control, where the system being controlled is a portfolio consisting of multiple investment components, and the control is its component weights. Consequently, the problem could be solved using model-free Reinforcement Learning (RL) without knowing specific component dynamics. By examining existing methods of both value-based and policy-based model-free RL for the portfolio optimization problem, we identify some of the key unresolved questions and difficulties facing today's portfolio managers of applying model-free RL to their investment portfolios.

Explore related subjects

q-fin.PM cs.AI cs.LG stat.ML

Keep this discovery

BibTeX RIS

Yoshiharu Sato. 2019-04-10. Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey. https://arxiv.org/abs/1904.04973

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Entropic Value-at-Risk parity for tempered stable returns

We develop Entropic Value-at-Risk (EVaR) parity for tempered stable returns. EVaR-based inverse risk parity (IRP) and equal risk contribution (ERC) portfolios are constructed using multivariate normal tempered stable models and independent component analysis with tempered stable components. We derive the corresponding asset-level EVaR and EVaR-deviation contributions and use the latter to separate the fitted location term from EVaR risk contributions. Under Gaussian returns, EVaR-deviation IRP and ERC recover conventional volatility IRP and ERC weights. We evaluate the resulting portfolios in three investment universes. Empirically, EVaR-based ERC portfolios achieve positive Sharpe differences relative to equal weight across the universes.

q-fin.PM↗

arXiv2026

Simple Dynamic Stock/Bond/Gold Portfolios

For more than four decades, the 60/40 stock/bond portfolio has served as a benchmark for delivering reasonable returns without excessive risk. More recently, a 50/30/20 stock/bond/alternative portfolio has been suggested. We use gold as the alternative and as an inflation hedge. In this paper we ask: how much improvement over these benchmark fixed-weight portfolios can be obtained using widely available public data and standard methods from quantitative finance? We restrict ourselves to long-only dynamic portfolios of stocks, bonds, and gold, plus cash, rebalancing monthly, using only publicly available data. We evaluate portfolios on the conventional metrics: return, volatility, Sharpe ratio (computed in excess of the federal funds rate), drawdown, and turnover, in addition to consistency of performance over time, judged by the consistency of the realized annual volatility. Over the 20--year period 2006--2026, using a conservative estimate of trading costs, we show that all risk-adjusted and drawdown metrics are improved using simple volatility control, where we dynamically mix the fixed-weight portfolios with cash so as to target a fixed volatility. This method relies on a simple estimate of portfolio volatility derived from past returns. We also demonstrate that more sophisticated portfolios based on convex optimization---similar to those used in quantitative hedge funds---yield further substantial improvement in return and risk-adjusted return. We consider two such portfolios, one that uses a simple estimate of future returns based on past returns, and one that forecasts future returns based on past returns and just a handful of widely available public economic data. These portfolios also outperform a suite of standard risk-based allocation methods, such as risk parity and minimum variance, evaluated on the same assets and data.

q-fin.PM↗

arXiv2026

An Entropic Factor Model for Robust Portfolio Replication

Portfolio replication, or the construction of a tradable basket of assets to match the risk-return profile of a target benchmark, is fundamentally an ill-posed inverse problem. When restricted to a subset of available assets, classical variance-minimizing models often yield unstable, over-leveraged portfolios highly vulnerable to market shocks. We propose a unified, two-stage methodology rooted in information theory to achieve robust portfolio replication. First, we model the constituent asset returns against target factors, estimating parameters within data-driven empirical bounds via an entropy minimization principle. Second, using the same entropic approach, we determine the optimal weight replication. In both cases we use an entropy function of the Fermi-Dirac type defined directly on sets of constraints of the inverse problem. We validate this Entropic Factor Model (EFM) against standard Ordinary Least Squares (OLS) across five numerical experiments, including standard equity tracking, multi-asset synthesis, and severe stress-test scenarios. Empirical results demonstrate that the EFM consistently outperforms OLS in terms of annualized turnover and net-of-fees returns. Crucially, during the COVID-19 market crash and under severe idiosyncratic data corruption, the entropic framework acts as a probabilistic ``circuit breaker", defensively reducing capital allocation to compromised assets and providing a highly robust, risk-averse solution for generalized portfolio replication.

q-fin.PM↗