arXiv subjects

Julius Luy

Publications and source records attributed to Julius Luy.

3 recordsLinked to original sources

arXiv2024

Multi-Agent Soft Actor-Critic with Coordinated Loss for Autonomous Mobility-on-Demand Fleet Control

We study a sequential decision-making problem for a profit-maximizing operator of an autonomous mobility-on-demand system. Optimizing a central operator's vehicle-to-request dispatching policy requires efficient and effective fleet control strategies. To this end, we employ a multi-agent Soft Actor-Critic algorithm combined with weighted bipartite matching. We propose a novel vehicle-based algorithm architecture and adapt the critic's loss function to appropriately consider coordinated actions. Furthermore, we extend our algorithm to incorporate rebalancing capabilities. Through numerical experiments, we show that our approach outperforms state-of-the-art benchmarks by up to 12.9% for dispatching and up to 38.9% with integrated rebalancing.

eess.SY↗

arXiv2023

Strategic Workforce Planning in Crowdsourced Delivery with Hybrid Driver Fleets

Nowadays, logistics service providers (LSPs) increasingly consider using a crowdsourced workforce on the last mile to fulfill customers' expectations regarding same-day or on-demand delivery at reduced costs. The crowdsourced workforce's availability is, however, uncertain. Therefore, LSPs often hire additional fixed employees to perform deliveries when the availability of crowdsourced drivers is low. In this context, the reliability versus flexibility trade-off which LSPs face over a longer period, e.g., a year, remains unstudied. Against this background, we jointly study a workforce planning problem that considers fixed drivers (FDs) and the temporal development of the crowdsourced driver (CD) fleet over a long-term time horizon. We consider two types of CDs, gigworkers (GWs) and occasional drivers (ODs). While GWs are not sensitive to the request's destination and typically exhibit high availability, ODs only serve requests whose origin and destination coincide with their own private route's origin and destination. Moreover, to account for time horizon-specific dynamics, we consider stochastic turnover for both FDs and CDs as well as stochastic CD fleet growth. We formulate the resulting workforce planning problem as a Markov decision process (MDP) whose reward function reflects total costs, i.e., wages and operational costs arising from serving demand with FDs and CDs, and solve it via approximate dynamic programming (ADP). Applying our approach to an environment based on real-world demand data from GrubHub, we find that in fleets consisting of FDs and CDs, ADP-based hiring policies can outperform myopic hiring policies by up to 19% in total costs. In the studied setting, we observed that GWs reduce the LSP's total costs more than ODs. When we account for CDs' increased resignation probability when not being matched with enough requests, the amount of required FDs increases.

eess.SY↗

arXiv2023

Dynamic Neighborhood Construction for Structured Large Discrete Action Spaces

Large discrete action spaces (LDAS) remain a central challenge in reinforcement learning. Existing solution approaches can handle unstructured LDAS with up to a few million actions. However, many real-world applications in logistics, production, and transportation systems have combinatorial action spaces, whose size grows well beyond millions of actions, even on small instances. Fortunately, such action spaces exhibit structure, e.g., equally spaced discrete resource units. With this work, we focus on handling structured LDAS (SLDAS) with sizes that cannot be handled by current benchmarks: we propose Dynamic Neighborhood Construction (DNC), a novel exploitation paradigm for SLDAS. We present a scalable neighborhood exploration heuristic that utilizes this paradigm and efficiently explores the discrete neighborhood around the continuous proxy action in structured action spaces with up to $10^{73}$ actions. We demonstrate the performance of our method by benchmarking it against three state-of-the-art approaches designed for large discrete action spaces across two distinct environments. Our results show that DNC matches or outperforms state-of-the-art approaches while being computationally more efficient. Furthermore, our method scales to action spaces that so far remained computationally intractable for existing methodologies.

cs.LG↗