arXiv subjects

Maximilian Schiffer

Publications and source records attributed to Maximilian Schiffer.

At least 37 records · Page 2Linked to original sources

arXiv2025

Multi-Agent Soft Actor-Critic with Coordinated Loss for Autonomous Mobility-on-Demand Fleet Control

We study a sequential decision-making problem for a profit-maximizing operator of an autonomous mobility-on-demand system. Optimizing a central operator's vehicle-to-request dispatching policy requires efficient and effective fleet control strategies. To this end, we employ a multi-agent Soft Actor-Critic algorithm combined with weighted bipartite matching. We propose a novel vehicle-based algorithm architecture and adapt the critic's loss function to appropriately consider coordinated actions. Furthermore, we extend our algorithm to incorporate rebalancing capabilities. Through numerical experiments, we show that our approach outperforms state-of-the-art benchmarks by up to 12.9% for dispatching and up to 38.9% with integrated rebalancing.

eess.SY↗

arXiv2025

Infrastructure Planning for Inductive Charging in Electrified Shuttle Systems

In response to climate goals, growing environmental awareness, and financial incentives, municipalities increasingly seek to electrify public transportation networks. We study the problem of locating stationary and dynamic inductive charging stations for electric vehicles (EVs), allowing detours from fixed transit routes and schedules. Dynamic charging, which enables energy transfer while driving, reduces space usage in dense urban areas and lowers vehicle idle times. We formulate a cost-minimization problem that considers both infrastructure and operational costs and propose an Iterated Local Search (ILS) algorithm to solve instances of realistic size. Each configuration requires solving a decomposed subproblem comprising multiple resource-constrained shortest-path problems. For this, we employ a bi-directional label-setting algorithm with lazy dominance checks based on local bounds. On adapted benchmark instances, our approach outperforms a commercial solver by up to 60% in solution quality. We further apply our method to a real-world case study in Hof, Germany. Results indicate that, under current cost structures calibrated from a test track in Bad Staffelstein, dynamic inductive charging is not yet cost-competitive with stationary alternatives. We quantify the value of allowing detours at up to 3.5% of the total system cost and show that integrating photovoltaics with decentralized energy storage can yield savings exceeding 20%.

math.OC↗

arXiv2025

Exploring Competitive and Collusive Behaviors in Algorithmic Pricing with Deep Reinforcement Learning

Nowadays, a significant share of the business-to-consumer sector is based on online platforms like Amazon and Alibaba and uses AI for pricing strategies. This has sparked debate on whether pricing algorithms may tacitly collude to set supra-competitive prices without being explicitly designed to do so. Our study addresses these concerns by examining the risk of collusion when Reinforcement Learning (RL) algorithms are used to decide on pricing strategies in competitive markets. Prior research in this field focused on Tabular Q-learning (TQL) and led to opposing views on whether learning-based algorithms can result in supra-competitive prices. Building on this, our work contributes to this ongoing discussion by providing a more nuanced numerical study that goes beyond TQL, additionally capturing off- and on- policy Deep Reinforcement Learning (DRL) algorithms, two distinct families of DRL algorithms that recently gained attention for algorithmic pricing. We study multiple Bertrand oligopoly variants and show that algorithmic collusion depends on the algorithm used. In our experiments, we observed that TQL tends to exhibit higher collusion and price dispersion. Moreover, it suffers from instability and disparity, as agents with higher learning rates consistently achieve higher profits, and it lacks robustness in state representation, with pricing dynamics varying significantly based on information access. In contrast, DRL algorithms, such as PPO and DQN, generally converge to lower prices closer to the Nash equilibrium. Additionally, we show that when pre-trained TQL agents interact with DRL agents, the latter quickly outperforms the former, highlighting the advantages of DRL in pricing competition. Lastly, we find that competition between heterogeneous DRL algorithms, such as PPO and DQN, tends to reduce the likelihood of supra-competitive pricing.

econ.GN↗

arXiv2025

Optimization-Augmented Machine Learning for Vehicle Operations in Emergency Medical Services

Minimizing response times to meet legal requirements and serve patients in a timely manner is crucial for Emergency Medical Service (EMS) systems. Achieving this goal necessitates optimizing operational decision-making to efficiently manage ambulances. Against this background, we study a centrally controlled EMS system for which we learn an online ambulance dispatching and redeployment policy that aims at minimizing the mean response time of ambulances within the system by dispatching an ambulance upon receiving an emergency call and redeploying it to a waiting location upon the completion of its service. We propose a novel combinatorial optimization-augmented machine learning pipeline that allows to learn efficient policies for ambulance dispatching and redeployment. In this context, we further show how to solve the underlying full-information problem to generate training data and propose an augmentation scheme that improves our pipeline's generalization performance by mitigating a possible distribution mismatch with respect to the considered state space. Compared to existing methods that rely on augmentation during training, our approach offers substantial runtime savings of up to 87.9% while yielding competitive performance. To evaluate the performance of our pipeline against current industry practices, we conduct a numerical case study on the example of San Francisco's 911 call data. Results show that the learned policies outperform the online benchmarks across various resource and demand scenarios, yielding a reduction in mean response time of up to 30%.

cs.LG↗

arXiv2025

Operational route planning under uncertainty for Demand Adaptive Systems

With an increasing need for more flexible mobility services, we consider an operational problem arising in the planning of Demand Adaptive Systems (DAS). Motivated by the decision of whether to accept or reject passenger requests in real time in a DAS, we introduce the operational route planning problem of DASs. To this end, we propose an algorithmic framework that allows an operator to plan which passengers to serve in a DAS in real-time. To do so, we model the operational route planning problem as a Markov decision process (MDP) and utilize a rolling horizon approach to approximate the MDP via a two-stage stochastic program in each timestep to decide on the next action. Furthermore, we determine the deterministic equivalent of our approximation through sample-based approximation. This allows us to decompose the deterministic equivalent of our two-stage stochastic program into several full information planning problems, which can be solved in parallel efficiently. Additionally, we propose a consensus-based heuristic and a myopic approach. We perform extensive numerical studies based on real-world data provided to us by the public transportation provider of Munich, Germany. We show that our exact decomposition yields the best results in under five seconds, and our heuristic approach reduces the serial computation time by 17 - 57% compared to our exact decomposition, with a solution quality decline of less than one percent. From a managerial perspective, we show that by switching a fixed-line bus route to a DAS, an operator can increase profit by up to 49% and the number of served passengers by up to 35% while only increasing the travel distance of the bus by 14%. Furthermore, we show that an operator can reduce their cost per passenger by 43 - 51% by increasing route flexibility and that incentivizing passengers to walk slightly longer distances reduces the cost per passenger by 83-85%.

math.OC↗

arXiv2025

Solving Large-Scale Two-Echelon Location Routing Problems in City Logistics

Logistic service providers increasingly focus on two-echelon distribution systems to efficiently manage thousands of deliveries in urban environments. Effectively operating such systems requires designing cost-efficient delivery networks while addressing the challenges of increasing e-commerce demands. In this context, we focus on a two-echelon location routing problem with mobile depots and direct shipment, where decisions involve locating micro-depots, and designing first and second-level routes. Our model also incorporates the flexibility of direct shipments from the main depot to customers. To solve such large-scale problems efficiently, we propose a metaheuristic approach that integrates a set cover problem with an adaptive large neighborhood search (ALNS). Our ALNS approach generates a set of promising routes and micro-depot locations using destroy and repair operators while using a local search for intensification. We then utilize the set cover problem to find better network configurations. Additionally, we present a decomposition-based cluster-first, route-second approach to solve large-scale instances efficiently. We show the efficacy of our algorithm on well-known benchmark datasets and provide managerial insights based on a case study for the city of Munich. Our decomposition approach provides comparable results while reducing computational times by a factor of 15. Our case study results show that allowing direct shipment can reduce total costs by 4.7% and emissions by 11%, while increasing truck utilizations by 42%. We find that integrating both stationary and mobile micro-depots, along with allowing direct shipments, can reduce total costs by 5.9% compared to traditional two-echelon delivery structures.

math.OC↗

arXiv2025

Preference-aware compensation policies for crowdsourced on-demand services

Crowdsourced on-demand services offer benefits such as reduced costs, faster service fulfillment times, greater adaptability, and contributions to sustainable urban transportation in on-demand delivery contexts. However, the success of an on-demand platform that utilizes crowdsourcing relies on finding a compensation policy that strikes a balance between creating attractive offers for gig workers and ensuring profitability. In this work, we examine a dynamic pricing problem for an on-demand platform that sets request-specific compensation of gig workers in a discrete-time framework, where requests and workers arrive stochastically. The operator's goal is to determine a compensation policy that maximizes the total expected reward over the time horizon. Our approach introduces compensation strategies that explicitly account for gig worker request preferences. To achieve this, we employ the Multinomial Logit model to represent the acceptance probabilities of gig workers, and, as a result, derive an analytical solution that utilizes post-decision states. Subsequently, we integrate this solution into an approximate dynamic programming algorithm. We compare our algorithm against benchmark algorithms, including formula-based policies and an upper bound provided by the full information linear programming solution. Our algorithm demonstrates consistent performance across diverse settings, achieving improvements of at least 2.5-7.5% in homogeneous gig worker populations and 9% in heterogeneous populations over benchmarks, based on fully synthetic data. For real-world data, it surpasses benchmarks by 8% in weak and 20% in strong location preference scenarios.

cs.LG↗

arXiv2024

WardropNet: Traffic Flow Predictions via Equilibrium-Augmented Learning

When optimizing transportation systems, anticipating traffic flows is a central element. Yet, computing such traffic equilibria remains computationally expensive. Against this background, we introduce a novel combinatorial optimization augmented neural network architecture that allows for fast and accurate traffic flow predictions. We propose WardropNet, a neural network that combines classical layers with a subsequent equilibrium layer: the first ones inform the latter by predicting the parameterization of the equilibrium problem's latency functions. Using supervised learning we minimize the difference between the actual traffic flow and the predicted output. We show how to leverage a Bregman divergence fitting the geometry of the equilibria, which allows for end-to-end learning. WardropNet outperforms pure learning-based approaches in predicting traffic equilibria for realistic and stylized traffic scenarios. On realistic scenarios, WardropNet improves on average for time-invariant predictions by up to 72% and for time-variant predictions by up to 23% over pure learning-based approaches.

cs.LG↗

arXiv2024

Algorithmic Collusion in Dynamic Pricing with Deep Reinforcement Learning

Nowadays, a significant share of the Business-to-Consumer sector is based on online platforms like Amazon and Alibaba and uses Artificial Intelligence for pricing strategies. This has sparked debate on whether pricing algorithms may tacitly collude to set supra-competitive prices without being explicitly designed to do so. Our study addresses these concerns by examining the risk of collusion when Reinforcement Learning algorithms are used to decide on pricing strategies in competitive markets. Prior research in this field focused on Tabular Q-learning (TQL) and led to opposing views on whether learning-based algorithms can lead to supra-competitive prices. Our work contributes to this ongoing discussion by providing a more nuanced numerical study that goes beyond TQL by additionally capturing off- and on-policy Deep Reinforcement Learning (DRL) algorithms. We study multiple Bertrand oligopoly variants and show that algorithmic collusion depends on the algorithm used. In our experiments, TQL exhibits higher collusion and price dispersion phenomena compared to DRL algorithms. We show that the severity of collusion depends not only on the algorithm used but also on the characteristics of the market environment. We further find that Proximal Policy Optimization appears to be less sensitive to collusive outcomes compared to other state-of-the-art DRL algorithms.

econ.GN↗

arXiv2024

Global Rewards in Multi-Agent Deep Reinforcement Learning for Autonomous Mobility on Demand Systems

We study vehicle dispatching in autonomous mobility on demand (AMoD) systems, where a central operator assigns vehicles to customer requests or rejects these with the aim of maximizing its total profit. Recent approaches use multi-agent deep reinforcement learning (MADRL) to realize scalable yet performant algorithms, but train agents based on local rewards, which distorts the reward signal with respect to the system-wide profit, leading to lower performance. We therefore propose a novel global-rewards-based MADRL algorithm for vehicle dispatching in AMoD systems, which resolves so far existing goal conflicts between the trained agents and the operator by assigning rewards to agents leveraging a counterfactual baseline. Our algorithm shows statistically significant improvements across various settings on real-world data compared to state-of-the-art MADRL algorithms with local rewards. We further provide a structural analysis which shows that the utilization of global rewards can improve implicit vehicle balancing and demand forecasting abilities. Our code is available at https://github.com/tumBAIS/GR-MADRL-AMoD.

cs.LG↗

arXiv2024

Staggered Routing in Autonomous Mobility-on-Demand Systems

In autonomous mobility-on-demand systems, effectively managing vehicle flows to mitigate induced congestion and ensure efficient operations is imperative for system performance and positive customer experience. Against this background, we study the potential of staggered routing, i.e., purposely delaying trip departures from a system perspective, in order to reduce congestion and ensure efficient operations while still meeting customer time windows. We formalize the underlying planning problem and show how to efficiently model it as a mixed integer linear program. Moreover, we present a matheuristic that allows us to efficiently solve large-scale real-world instances both in an offline full-information setting and its online rolling horizon counterpart. We conduct a numerical study for Manhattan, New York City, focusing on low- and highly-congested scenarios. Our results show that in low-congestion scenarios, staggering trip departures allows mitigating, on average, 94% of the induced congestion in a full information setting. In a rolling horizon setting, our algorithm allows us to reduce 90% of the induced congestion. In high-congestion scenarios, we observe an average reduction of 66% as the full information bound and an average reduction of 56% in our online setting. Surprisingly, we show that these reductions can be reached by shifting trip departures by a maximum of six minutes in both the low and high-congestion scenarios.

math.OC↗

arXiv2024

On the Impact of Co-Optimizing Station Locations, Trip Assignment, and Charging Schedules for Electric Buses

As many public transportation systems around the world transition to electric buses, the planning and operation of fleets can be improved via tailored decision-support tools. In this work, we study the impact of jointly locating charging facilities, assigning electric buses to trips, and determining when and where to charge the buses. We propose a mixed integer linear program that co-optimizes planning and operational decisions jointly and an iterated local search heuristic to solve large-scale instances. Herein, we use a concurrent scheduler algorithm to generate an initial feasible solution, which serves as a starting point for our iterated local search algorithm. In the sequential case, we first optimize trip assignments and charging locations. Charging schedules are then determined after fixing the optimal decisions from the first level. The joint model, on the other hand, integrates charge scheduling within the local search procedure. The solution quality of the joint and sequential iterated local search models are compared for multiple real-world bus transit networks. Our results demonstrate that joint models can help further improve operating costs by 14.1% and lower total costs by about 4.1% on average compared with sequential models. In addition, energy consumption costs and contracted power capacity costs have been reduced significantly due to our integrated planning approach.

math.OC↗

arXiv2024

Dynamic Neighborhood Construction for Structured Large Discrete Action Spaces

Large discrete action spaces (LDAS) remain a central challenge in reinforcement learning. Existing solution approaches can handle unstructured LDAS with up to a few million actions. However, many real-world applications in logistics, production, and transportation systems have combinatorial action spaces, whose size grows well beyond millions of actions, even on small instances. Fortunately, such action spaces exhibit structure, e.g., equally spaced discrete resource units. With this work, we focus on handling structured LDAS (SLDAS) with sizes that cannot be handled by current benchmarks: we propose Dynamic Neighborhood Construction (DNC), a novel exploitation paradigm for SLDAS. We present a scalable neighborhood exploration heuristic that utilizes this paradigm and efficiently explores the discrete neighborhood around the continuous proxy action in structured action spaces with up to $10^{73}$ actions. We demonstrate the performance of our method by benchmarking it against three state-of-the-art approaches designed for large discrete action spaces across two distinct environments. Our results show that DNC matches or outperforms state-of-the-art approaches while being computationally more efficient. Furthermore, our method scales to action spaces that so far remained computationally intractable for existing methodologies.

cs.LG↗

arXiv2024

Learning-based Online Optimization for Autonomous Mobility-on-Demand Fleet Control

Autonomous mobility-on-demand systems are a viable alternative to mitigate many transportation-related externalities in cities, such as rising vehicle volumes in urban areas and transportation-related pollution. However, the success of these systems heavily depends on efficient and effective fleet control strategies. In this context, we study online control algorithms for autonomous mobility-on-demand systems and develop a novel hybrid combinatorial optimization enriched machine learning pipeline which learns online dispatching and rebalancing policies from optimal full-information solutions. We test our hybrid pipeline on large-scale real-world scenarios with different vehicle fleet sizes and various request densities. We show that our approach outperforms state-of-the-art greedy, and model-predictive control approaches with respect to various KPIs, e.g., by up to 17.1% and on average by 6.3% in terms of realized profit.

math.OC↗

arXiv2024

Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts

We study the robustness of deep reinforcement learning algorithms against distribution shifts within contextual multi-stage stochastic combinatorial optimization problems from the operations research domain. In this context, risk-sensitive algorithms promise to learn robust policies. While this field is of general interest to the reinforcement learning community, most studies up-to-date focus on theoretical results rather than real-world performance. With this work, we aim to bridge this gap by formally deriving a novel risk-sensitive deep reinforcement learning algorithm while providing numerical evidence for its efficacy. Specifically, we introduce discrete Soft Actor-Critic for the entropic risk measure by deriving a version of the Bellman equation for the respective Q-values. We establish a corresponding policy improvement result and infer a practical algorithm. We introduce an environment that represents typical contextual multi-stage stochastic combinatorial optimization problems and perform numerical experiments to empirically validate our algorithm's robustness against realistic distribution shifts, without compromising performance on the training distribution. We show that our algorithm is superior to risk-neutral Soft Actor-Critic as well as to two benchmark approaches for robust deep reinforcement learning. Thereby, we provide the first structured analysis on the robustness of reinforcement learning under distribution shifts in the realm of contextual multi-stage stochastic combinatorial optimization problems.

cs.LG↗

arXiv2024

Contextual Stochastic Vehicle Routing with Time Windows

We study the vehicle routing problem with time windows (VRPTW) and stochastic travel times, in which the decision-maker observes related contextual information, represented as feature variables, before making routing decisions. Despite the extensive literature on stochastic VRPs, the integration of feature variables has received limited attention in this context. We introduce the contextual stochastic VRPTW, which minimizes the total transportation cost and expected late arrival penalties conditioned on the observed features. Since the joint distribution of travel times and features is unknown, we present novel data-driven prescriptive models that use historical data to provide an approximate solution to the problem. We distinguish the prescriptive models between point-based approximation, sample average approximation, and penalty-based approximation, each taking a different perspective on dealing with stochastic travel times and features. We develop specialized branch-price-and-cut algorithms to solve these data-driven prescriptive models. In our computational experiments, we compare the out-of-sample cost performance of different methods on instances with up to one hundred customers. Our results show that, surprisingly, a feature-dependent sample average approximation outperforms existing and novel methods in most settings.

cs.LG↗

arXiv2023

On the Interplay between Self-Driving Cars and Public Transportation

Cities worldwide struggle with overloaded transportation systems and their externalities. The emerging autonomous transportation technology has the potential to alleviate these issues, but the decisions of profit-maximizing operators running large autonomous fleets could negatively impact other stakeholders and the transportation system. An analysis of these tradeoffs requires modeling the modes of transportation in a unified framework. In this paper, we propose such a framework, which allows us to study the interplay among mobility service providers (MSPs), public transport authorities, and customers. Our framework combines a graph-theoretic network model for the transportation system with a game-theoretic market model in which MSPs are profit maximizers while customers select individually-optimal transportation options. We apply our framework to data for the city of Berlin and present sensitivity analyses to study parameters that MSPs or municipalities can strategically influence. We show that autonomous ride-hailing systems may cannibalize a public transportation system, serving between 7% and 80% of all customers, depending on market conditions and policy restrictions.

eess.SY↗

arXiv2023

Designing Tractable Piecewise Affine Policies for Multi-Stage Adjustable Robust Optimization

We study piecewise affine policies for multi-stage adjustable robust optimization (ARO) problems with non-negative right-hand side uncertainty. First, we construct new dominating uncertainty sets and show how a multi-stage ARO problem can be solved efficiently with a linear program when uncertainty is replaced by these new sets. We then demonstrate how solutions for this alternative problem can be transformed into solutions for the original problem. By carefully choosing the dominating sets, we prove strong approximation bounds for our policies and extend many previously best-known bounds for the two-staged problem variant to its multi-stage counterpart. Moreover, the new bounds are - to the best of our knowledge - the first bounds shown for the general multi-stage ARO problem considered. We extensively compare our policies to other policies from the literature and prove relative performance guarantees. In two numerical experiments, we identify beneficial and disadvantageous properties for different policies and present effective adjustments to tackle the most critical disadvantages of our policies. Overall, the experiments show that our piecewise affine policies can be computed by orders of magnitude faster than affine policies, while often yielding comparable or even better results.

math.OC↗