arXiv subjects

Nicole Bäuerle

Publications and source records attributed to Nicole Bäuerle.

At least 19 recordsLinked to original sources

arXiv2026

Policy stability and ultimate stationarity in discounted risk-sensitive stochastic control

We study discrete-time Markov Decision Processes (MDPs) on finite state-action spaces and analyze the stability of optimal policies and value functions in the long-run discounted risk-sensitive objective setting. Our analysis addresses robustness with respect to perturbations of the risk-aversion parameter and the discount factor, the emergence of ultimate stationarity, and the interaction between discounted and averaged formulations under suitable mixing assumptions. We further investigate limiting regimes associated with vanishing discount and vanishing risk sensitivity, and discuss the role of Blackwell-type stability properties in the discounted setting. Finally, we provide numerical illustrations that highlight the intrinsic non-stationarity of optimal discounted risk-sensitive policies.

math.OC↗

arXiv2026

Mean-Variance Optimization in Ambiguous Financial Markets with Learning

We consider a continuous time investment problem in a multi-asset Black-Scholes market with the following features: The assets' drifts are not known and constitute a source of model ambiguity. However, there is a prior distribution (knowledge) on the possible drifts. Our investor is ambiguity averse and wants to maximize a mean-variance criterion for the terminal wealth where ambiguity aversion is incorporated in a smooth way. We consider here the criterion introduced in Maccheroni et al. 2013 where the variance is decomposed and each part is weighted differently to account for different levels of market risk and model ambiguity aversion. We use a novel approach to find the optimal dynamic investment strategy within the class of all adapted strategies which allow for learning. We also present a number of numerical results which help to understand how the model parameters affect the optimal investment strategy. In general it turns out that ambiguity averse investors invest less in the risky assets.

q-fin.PM↗

arXiv2026

Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descent

The goal of this paper is to analyze distributional Markov Decision Processes as a class of control problems in which the objective is to learn policies that steer the distribution of a cumulative reward toward a prescribed target law, rather than optimizing an expected value or a risk functional. To solve the resulting distributional control problem in a model-free setting, we propose a policy-gradient algorithm based on neural-network parameterizations of randomized Markov policies, defined on an augmented state space and a sample-based evaluation of the characteristic-function loss. Under mild regularity and growth assumptions, we prove convergence of the algorithm to stationary points using stochastic approximation techniques. Several numerical experiments illustrate the ability of the method to match complex target distributions, recover classical optimal policies when they exist, and reveal intrinsic non-uniqueness phenomena specific to distributional control.

math.OC↗

arXiv2025

Yet Another Distributional Bellman Equation

We consider non-standard Markov Decision Processes (MDPs) where the target function is not only a simple expectation of the accumulated reward. Instead, we consider rather general functionals of the joint distribution of terminal state and accumulated reward which have to be optimized. For finite state and compact action space, we show how to solve these problems by defining a lifted MDP whose state space is the space of distributions over the true states of the process. We derive a Bellman equation in this setting, which can be considered as a distributional Bellman equation. Well-known cases like the standard MDP and quantile MDPs are shown to be special examples of our framework. We also apply our model to a variant of an optimal transport problem.

math.OC↗

arXiv2025

Relative portfolio optimization via a value at risk based constraint

In this paper, we consider $n$ agents who invest in a general financial market that is free of arbitrage and complete. The aim of each investor is to maximize her expected utility while ensuring, with a specified probability, that her terminal wealth exceeds a benchmark defined by her competitors' performance. This setup introduces an interdependence between agents, leading to a search for Nash equilibria. In the case of two agents and CRRA utility, we are able to derive all Nash equilibria in terms of terminal wealth. For $n>2$ agents and logarithmic utility we distinguish two cases. In the first case, the probabilities in the constraint are small and we can characterize all Nash equilibria. In the second case, the probabilities are larger and we look for Nash equilibria in a certain set. We also discuss the impact of the competition using some numerical examples. As a by-product, we solve some portfolio optimization problems with probability constraints.

math.OC↗

arXiv2024

Blackwell optimality and policy stability for long-run risk sensitive stochastic control

This paper analyzes the stability of optimal policies in the long-run stochastic control framework with an averaged risk-sensitive criterion for discrete-time MDPs on finite state-action space. In particular, we study the robustness of optimal controls when perturbations to the risk-aversion parameter are applied, and investigate the Blackwell property, together with its link to the risk-sensitive vanishing discount approximation framework. Finally, we present examples that help to better understand the intricacies of the risk-sensitive control framework.

math.OC↗

arXiv2024

Continuous-time mean field Markov decision models

We consider a finite number of $N$ statistically equal agents, each moving on a finite set of states according to a continuous-time Markov Decision Process (MDP). Transition intensities of the agents and generated rewards depend not only on the state and action of the agent itself, but also on the states of the other agents as well as the chosen action. Interactions like this are typical for a wide range of models in e.g. biology, epidemics, finance, social science and queueing systems among others. The aim is to maximize the expected discounted reward of the system, i.e. the agents have to cooperate as a team. Computationally this is a difficult task when $N$ is large. Thus, we consider the limit for $N\to\infty.$ In contrast to other papers we treat this problem from an MDP perspective. This has the advantage that we need less regularity assumptions in order to construct asymptotically optimal strategies than using viscosity solutions of HJB equations. The convergence rate is $1/\sqrt{N}$. We show how to apply our results using two examples: a machine replacement problem and a problem from epidemics. We also show that optimal feedback policies from the limiting problem are not necessarily asymptotically optimal.

math.PR↗

arXiv2024

Nash equilibria for relative investors with (non)linear price impact

We consider the strategic interaction of $n$ investors who are able to influence a stock price process and at the same time measure their utilities relative to the other investors. Our main aim is to find Nash equilibrium investment strategies in this setting in a financial market driven by a Brownian motion and investigate the influence the price impact has on the equilibrium. We consider both CRRA and CARA utility functions. Our findings show that the problem is well-posed as long as the price impact is at most linear. Moreover, numerical results reveal that the investors behave very aggressively when the price impact is beyond a critical parameter.

math.OC↗

arXiv2024

Optimal investment in ambiguous financial markets with learning

We consider the classical multi-asset Merton investment problem under drift uncertainty, i.e. the asset price dynamics are given by geometric Brownian motions with constant but unknown drift coefficients. The investor assumes a prior drift distribution and is able to learn by observing the asset prize realizations during the investment horizon. While the solution of an expected utility maximizing investor with constant relative risk aversion (CRRA) is well known, we consider the optimization problem under risk and ambiguity preferences by means of the KMM (Klibanoff et al. (2005)) approach. Here, the investor maximizes a double certainty equivalent. The inner certainty equivalent is for given drift coefficient, the outer is based on a drift distribution. Assuming also a CRRA type ambiguity function, it turns out that the optimal strategy can be stated in terms of the solution without ambiguity preferences but an adjusted drift distribution. To the best of our knowledge an explicit solution method in this setting is new. We rely on some duality theorems to prove our statements. Based on our theoretical results, we are able to shed light on the impact of the prior drift distribution as well as the consequences of ambiguity preferences via the transfer to an adjusted drift distribution, i.e. we are able to explain the interaction of risk and ambiguity preferences. We compare our results with the ones in a pre-commitment setup where the investor is restricted to deterministic strategies. It turns out that (under risk and ambiguity aversion) an infinite investment horizon implies in both cases a maximin decision rule, i.e. the investor follows the worst (best) Merton fraction (over all realizations of it) if she is more (less) risk averse than a log-investor. We illustrate our findings with an extensive numerical study.

q-fin.PM↗

arXiv2023

Markov Decision Processes with Risk-Sensitive Criteria: An Overview

The paper provides an overview of the theory and applications of risk-sensitive Markov decision processes. The term 'risk-sensitive' refers here to the use of the Optimized Certainty Equivalent as a means to measure expectation and risk. This comprises the well-known entropic risk measure and Conditional Value-at-Risk. We restrict our considerations to stationary problems with an infinite time horizon. Conditions are given under which optimal policies exist and solution procedures are explained. We present both the theory when the Optimized Certainty Equivalent is applied recursively as well as the case where it is applied to the cumulated reward. Discounted as well as non-discounted models are reviewed

q-fin.RM↗

arXiv2023

Optimal investment under partial information and robust VaR-type constraint

This paper extends the utility maximization literature by combining partial information and (robust) regulatory constraints. Partial information is characterized by the fact that the stock price itself is observable by the optimizing financial institution, but the outcome of the market price of the risk $θ$ is unknown to the institution. The regulator develops either a congruent or distinct perception of the market price of risk in comparison to the financial institution when imposing the Value-at-Risk (VaR) constraint. We also discuss a robust VaR constraint in which the regulator uses a worst-case measure. The solution to our optimization problem takes the same form as in the full information case: optimal wealth can be expressed as a decreasing function of state price density. The optimal wealth is equal to the minimum regulatory financing requirement in the intermediate economic states. The key distinction lies in the fact that the price density in the final state depends on the overall evolution of the estimated market price of risk, denoted as $\hatθ(s)$ or that the upper boundary of the intermediate region exhibits stochastic behavior.

q-fin.RM↗

arXiv2023

Time-consistency in the mean-variance problem: A new perspective

We investigate discrete-time mean-variance portfolio selection problems viewed as a Markov decision process. We transform the problems into a new model with deterministic transition function for which the Bellman optimality equation holds. In this way, we can solve the problem recursively and obtain a time-consistent solution, that is an optimal solution that meets the Bellman optimality principle. We apply our technique for solving explicitly a more general framework.

math.OC↗

arXiv2022

Nash equilibria for relative investors via no-arbitrage arguments

Within a common arbitrage-free semimartingale financial market we consider the problem of determining all Nash equilibrium investment strategies for $n$ agents who try to maximize the expected utility of their relative wealth. The utility function can be rather general here. Exploiting the linearity of the stochastic integral and making use of the classical pricing theory we are able to express all Nash equilibrium investment strategies in terms of the optimal strategies for the classical one agent expected utility problems. The corresponding mean field problem is solved in the same way. We give four applications of specific financial markets and compare our results with those given in the literature.

math.OC↗

arXiv2022

Mean Field Markov Decision Processes

We consider mean-field control problems in discrete time with discounted reward, infinite time horizon and compact state and action space. The existence of optimal policies is shown and the limiting mean-field problem is derived when the number of individuals tends to infinity. Moreover, we consider the average reward problem and show that the optimal policy in this mean-field limit is $\varepsilon$-optimal for the discounted problem if the number of individuals is large and the discount factor close to one. This result is very helpful, because it turns out that in the special case when the reward does only depend on the distribution of the individuals, we obtain a very interesting subclass of problems where an average reward optimal policy can be obtained by first computing an optimal measure from a static optimization problem and then achieving it with Markov Chain Monte Carlo methods. We give two applications: Avoiding congestion an a graph and optimal positioning on a market place which we solve explicitly.

math.OC↗

arXiv2021

Bayesian optimal investment and reinsurance with dependent financial and insurance risks

Major events like natural catastrophes or the COVID-19 crisis have impact both on the financial market and on claim arrival intensities and claim sizes of insurers. Thus, when optimal investment and reinsurance strategies have to be determined it is important to consider models which reflect this dependence. In this paper we make a proposal how to generate dependence between the financial market and claim sizes in times of crisis and determine via a stochastic control approach an optimal investment and reinsurance strategy which maximizes the expected exponential utility of terminal wealth. Moreover, we also allow that the claim size distribution may be learned in the model. We give comparisons and bounds on the optimal strategy using simple models. What turns out to be very surprising is that numerical results indicate that even a minimal dependence which is created in this model has a huge impact on the control in the sense that the insurer is much more prudent then.

q-fin.PM↗

arXiv2020

Minimizing Spectral Risk Measures Applied to Markov Decision Processes

We study the minimization of a spectral risk measure of the total discounted cost generated by a Markov Decision Process (MDP) over a finite or infinite planning horizon. The MDP is assumed to have Borel state and action spaces and the cost function may be unbounded above. The optimization problem is split into two minimization problems using an infimum representation for spectral risk measures. We show that the inner minimization problem can be solved as an ordinary MDP on an extended state space and give sufficient conditions under which an optimal policy exists. Regarding the infinite dimensional outer minimization problem, we prove the existence of a solution and derive an algorithm for its numerical approximation. Our results include the findings in Bäuerle and Ott (2011) in the special case that the risk measure is Expected Shortfall. As an application, we present a dynamic extension of the classical static optimal reinsurance problem, where an insurance company minimizes its cost of capital.

math.OC↗

arXiv2020

Stochastic dynamic programming with non-linear discounting

In this paper, we study a Markov decision process with a non-linear discount function and with a Borel state space. We define a recursive discounted utility, which resembles non-additive utility functions considered in a number of models in economics. Non-additivity here follows from non-linearity of the discount function. Our study is complementary to the work of Jaśkiewicz, Matkowski and Nowak (Math. Oper. Res. 38 (2013), 108-121), where also non-linear discounting is used in the stochastic setting, but the expectation of utilities aggregated on the space of all histories of the process is applied leading to a non-stationary dynamic programming model. Our aim is to prove that in the recursive discounted utility case the Bellman equation has a solution and there exists an optimal stationary policy for the problem in the infinite time horizon. Our approach includes two cases: $(a)$ when the one-stage utility is bounded on both sides by a weight function multiplied by some positive and negative constants, and $(b)$ when the one-stage utility is unbounded from below.

math.OC↗

arXiv2020

Markov Decision Processes with Recursive Risk Measures

In this paper, we consider risk-sensitive Markov Decision Processes (MDPs) with Borel state and action spaces and unbounded cost under both finite and infinite planning horizons. Our optimality criterion is based on the recursive application of static risk measures. This is motivated by recursive utilities in the economic literature, has been studied before for the entropic risk measure and is extended here to an axiomatic characterization of suitable risk measures. We derive a Bellman equation and prove the existence of Markovian optimal policies. For an infinite planning horizon, the model is shown to be contractive and the optimal policy to be stationary. Moreover, we establish a connection to distributionally robust MDPs, which provides a global interpretation of the recursively defined objective function. Monotone models are studied in particular.

math.OC↗