arXiv subjects

Tien Mai

Publications and source records attributed to Tien Mai.

At least 19 recordsLinked to original sources

arXiv2026

Learning Sequential Mobility Choice: A Review of Route and Activity Choice through Inverse Reinforcement and Imitation Learning

Route and activity choice are connected levels of a common sequential mobility decision problem: activity choice determines what people do, where, and when, while route choice governs how they move between activities. This review develops a unified framework connecting transportation choice modeling with inverse reinforcement learning (IRL) and imitation learning (IL). Under explicit assumptions, recursive logit, logit dynamic discrete choice, and maximum-entropy IRL share a soft Bellman representation, while trajectory occupancies and network flows satisfy related conservation laws. However, utility, reward, policy, occupancy, constraints, and observation errors remain different estimands with different behavioral and counterfactual interpretations. We review constrained and inverse-constrained learning, occupancy-ratio and DICE methods, incomplete and mixed-quality demonstrations, graph and sequence learning, transfer, data fusion, multi-agent choice, and large language models. Our central message is that machine learning adds the greatest value when embedded within a behaviorally disciplined framework: exact transitions enforce feasibility, structured rewards preserve interpretable trade-offs, observation models address heterogeneous data sources, and network or equilibrium solvers produce coherent system outcomes. Such hybrid models can improve scalability and prediction without sacrificing behavioral identification or policy relevance.

econ.EM↗

arXiv2026

Constrained Pricing under Finite Mixtures of Logit

The mixed logit model is a flexible and widely used demand model in pricing and revenue management. However, existing work on mixed-logit pricing largely focuses on unconstrained settings, limiting its applicability in practice where prices are subject to business or regulatory constraints. We study the constrained pricing problem under multinomial and mixed logit demand models. For the multinomial logit model, corresponding to a single customer segment, we show that the constrained pricing problem admits a polynomial-time approximation scheme (PTAS) via a reformulation based on exponential cone programming, yielding an $\varepsilon$-optimal solution in polynomial time. For finite mixed logit models with $T$ customer segments, we reformulate the problem as a bilinear exponential cone program with $O(T)$ bilinear terms. This structure enables a Branch-and-Bound algorithm whose complexity is exponential only in $T$. Consequently, constrained pricing under finite mixtures of logit admits a PTAS when the number of customer segments is bounded. Numerical experiments demonstrate strong performance relative to state-of-the-art baselines.

math.OC↗

arXiv2025

Constrained Assortment and Price Optimization under Generalized Nested Logit Models

We study assortment and price optimization under the generalized nested logit (GNL) model, one of the most general and flexible modeling frameworks in discrete choice modeling. Despite its modeling advantages, optimization under GNL is highly challenging: even the pure assortment problem is NP-hard, and existing approaches rely on approximation schemes or are limited to simple cardinality constraints. In this paper, we develop the first exact and near-exact algorithms for constrained assortment and joint assortment--pricing optimization (JAP) under GNL. Our approach reformulates the problem into bilinear and exponential-cone convex programs and exploits convexity, concavity, and submodularity properties to generate strong cutting planes within a Branch-and-Cut framework (B\&C). We further extend this framework to the mixed GNL (MGNL) model, capturing heterogeneous customer segments, and to JAP with discrete prices. For the continuous pricing case, we propose a near-exact algorithm based on piecewise-linear approximation (PWLA) that achieves arbitrarily high precision under general linear constraints. Extensive computational experiments demonstrate that our methods substantially outperform state-of-the-art approximation approaches in both solution quality and scalability. In particular, we are able to solve large-scale instances with up to 1000 products and 20 nests, and to obtain near-optimal solutions for continuous pricing problems with negligible optimality gaps. To the best of our knowledge, this work resolves several open problems in assortment and price optimization under GNL.

econ.GN↗

arXiv2025

Equilibrium-Constrained Estimation of Recursive Logit Choice Models

The recursive logit (RL) model provides a flexible framework for modeling sequential decision-making in transportation and choice networks, with important applications in route choice analysis, multiple discrete choice problems, and activity-based travel demand modeling. Despite its versatility, estimation of the RL model typically relies on nested fixed-point (NFXP) algorithms that are computationally expensive and prone to numerical instability. We propose a new approach that reformulates the maximum likelihood estimation problem as an optimization problem with equilibrium constraints, where both the structural parameters and the value functions are treated as decision variables. We further show that this formulation can be equivalently transformed into a conic optimization problem with exponential cones, enabling efficient solution using modern conic solvers such as MOSEK. Experiments on synthetic and real-world datasets demonstrate that our convex reformulation achieves accuracy comparable to traditional methods while offering significant improvements in computational stability and efficiency, thereby providing a practical and scalable alternative for recursive logit model estimation.

econ.EM↗

arXiv2025

Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning

We study the problem of online multi-agent reinforcement learning (MARL) in environments with sparse rewards, where reward feedback is not provided at each interaction but only revealed at the end of a trajectory. This setting, though realistic, presents a fundamental challenge: the lack of intermediate rewards hinders standard MARL algorithms from effectively guiding policy learning. To address this issue, we propose a novel framework that integrates online inverse preference learning with multi-agent on-policy optimization into a unified architecture. At its core, our approach introduces an implicit multi-agent reward learning model, built upon a preference-based value-decomposition network, which produces both global and local reward signals. These signals are further used to construct dual advantage streams, enabling differentiated learning targets for the centralized critic and decentralized actors. In addition, we demonstrate how large language models (LLMs) can be leveraged to provide preference labels that enhance the quality of the learned reward model. Empirical evaluations on state-of-the-art benchmarks, including MAMuJoCo and SMACv2, show that our method achieves superior performance compared to existing baselines, highlighting its effectiveness in addressing sparse-reward challenges in online MARL.

cs.LG↗

arXiv2025

On the Estimation of Multinomial Logit and Nested Logit Models: A Conic Optimization Approach

In this paper, we revisit parameter estimation for multinomial logit (MNL), nested logit (NL), and tree-nested logit (TNL) models through the framework of convex conic optimization. Traditional approaches typically solve the maximum likelihood estimation (MLE) problem using gradient-based methods, which are sensitive to step-size selection and initialization, and may therefore suffer from slow or unstable convergence. In contrast, we propose a novel estimation strategy that reformulates these models as conic optimization problems, enabling more robust and reliable estimation procedures. Specifically, we show that the MLE for MNL admits an equivalent exponential cone program (ECP). For NL and TNL, we prove that when the dissimilarity (scale) parameters are fixed, the estimation problem is convex and likewise reducible to an ECP. Leveraging these results, we design a two-stage procedure: an outer loop that updates the scale parameters and an inner loop that solves the ECP to update the utility coefficients. The inner problems are handled by interior-point methods with iteration counts that grow only logarithmically in the target accuracy, as implemented in off-the-shelf solvers (e.g., MOSEK). Extensive experiments across estimation instances of varying size show that our conic approach attains better MLE solutions, greater robustness to initialization, and substantial speedups compared to standard gradient-based MLE, particularly on large-scale instances with high-dimensional specifications and large choice sets. Our findings establish exponential cone programming as a practical and scalable alternative for estimating a broad class of discrete choice models.

econ.EM↗

arXiv2025

Constrained Recursive Logit for Route Choice Analysis

The recursive logit (RL) model has become a widely used framework for route choice modeling, but it suffers from a key limitation: it assigns nonzero probabilities to all paths in the network, including those that are unrealistic, such as routes exceeding travel time deadlines or violating energy constraints. To address this gap, we propose a novel Constrained Recursive Logit (CRL) model that explicitly incorporates feasibility constraints into the RL framework. CRL retains the main advantages of RL-no path sampling and ease of prediction-but systematically excludes infeasible paths from the universal choice set. The model is inherently non-Markovian; to address this, we develop a tractable estimation approach based on extending the state space, which restores the Markov property and enables estimation using standard value iteration methods. We prove that our estimation method admits a unique solution under positive discrete costs and establish its equivalence to a multinomial logit model defined over restricted universal path choice sets. Empirical experiments on synthetic and real networks demonstrate that CRL improves behavioral realism and estimation stability, particularly in cyclic networks.

econ.EM↗

arXiv2025

Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations

Offline imitation learning typically learns from expert and unlabeled demonstrations, yet often overlooks the valuable signal in explicitly undesirable behaviors. In this work, we study offline imitation learning from contrasting behaviors, where the dataset contains both expert and undesirable demonstrations. We propose a novel formulation that optimizes a difference of KL divergences over the state-action visitation distributions of expert and undesirable (or bad) data. Although the resulting objective is a DC (Difference-of-Convex) program, we prove that it becomes convex when expert demonstrations outweigh undesirable demonstrations, enabling a practical and stable non-adversarial training objective. Our method avoids adversarial training and handles both positive and negative demonstrations in a unified framework. Extensive experiments on standard offline imitation learning benchmarks demonstrate that our approach consistently outperforms state-of-the-art baselines.

cs.LG↗

arXiv2025

MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrations

We study offline imitation learning (IL) in cooperative multi-agent settings, where demonstrations have unlabeled mixed quality - containing both expert and suboptimal trajectories. Our proposed solution is structured in two stages: trajectory labeling and multi-agent imitation learning, designed jointly to enable effective learning from heterogeneous, unlabeled data. In the first stage, we combine advances in large language models and preference-based reinforcement learning to construct a progressive labeling pipeline that distinguishes expert-quality trajectories. In the second stage, we introduce MisoDICE, a novel multi-agent IL algorithm that leverages these labels to learn robust policies while addressing the computational complexity of large joint state-action spaces. By extending the popular single-agent DICE framework to multi-agent settings with a new value decomposition and mixing architecture, our method yields a convex policy optimization objective and ensures consistency between global and local policies. We evaluate MisoDICE on multiple standard multi-agent RL benchmarks and demonstrate superior performance, especially when expert data is scarce.

cs.LG↗

arXiv2025

O-MAPL: Offline Multi-agent Preference Learning

Inferring reward functions from demonstrations is a key challenge in reinforcement learning (RL), particularly in multi-agent RL (MARL), where large joint state-action spaces and complex inter-agent interactions complicate the task. While prior single-agent studies have explored recovering reward functions and policies from human preferences, similar work in MARL is limited. Existing methods often involve separate stages of supervised reward learning and MARL algorithms, leading to unstable training. In this work, we introduce a novel end-to-end preference-based learning framework for cooperative MARL, leveraging the underlying connection between reward functions and soft Q-functions. Our approach uses a carefully-designed multi-agent value decomposition strategy to improve training efficiency. Extensive experiments on SMAC and MAMuJoCo benchmarks show that our algorithm outperforms existing methods across various tasks.

cs.LG↗

arXiv2025

Constrained Pricing in Choice-based Revenue Management

We consider a dynamic pricing problem in network revenue management where customer behavior is predicted by a choice model, i.e., the multinomial logit (MNL) model. The problem, even in the static setting (i.e., customer demand remains unchanged over time), is highly non-concave in prices. Existing studies mostly rely on the observation that the objective function is concave in terms of purchasing probabilities, implying that the static pricing problem with linear constraints on purchasing probabilities can be efficiently solved. However, this approach is limited in handling constraints on prices, noting that such constraints could be highly relevant in some real business considerations. To address this limitation, in this work, we consider a general pricing problem that involves constraints on both prices and purchasing probabilities. To tackle the non-concavity challenge, we develop an approximation mechanism that allows solving the constrained static pricing problem through bisection and mixed-integer linear programming (MILP). We further extend the approximation method to the dynamic pricing context. Our approach involves a resource decomposition method to address the curse of dimensionality of the dynamic problem, as well as a MILP approach to solving sub-problems to near-optimality. Numerical results based on generated instances of various sizes indicate the superiority of our approximation approach in both static and dynamic settings.

math.OC↗

arXiv2024

Competitive Facility Location with Market Expansion and Customer-centric Objective

We study a competitive facility location problem, where customer behavior is modeled and predicted using a discrete choice random utility model. The goal is to strategically place new facilities to maximize the overall captured customer demand in a competitive marketplace. In this work, we introduce two novel considerations. First, the total customer demand in the market is not fixed but is modeled as an increasing function of the customers' total utilities. Second, we incorporate a new term into the objective function, aiming to balance the firm's benefits and customer satisfaction. Our new formulation exhibits a highly nonlinear structure and is not directly solved by existing approaches. To address this, we first demonstrate that, under a concave market expansion function, the objective function is concave and submodular, allowing for a $(1-1/e)$ approximation solution by a simple polynomial-time greedy algorithm. We then develop a new method, called Inner-approximation, which enables us to approximate the mixed-integer nonlinear problem (MINLP), with arbitrary precision, by an MILP without introducing additional integer variables. We further demonstrate that our inner-approximation method consistently yields lower approximations than the outer-approximation methods typically used in the literature. Moreover, we extend our settings by considering a\textit{ general (non-concave)} market-expansion function and show that the Inner-approximation mechanism enables us to approximate the resulting MINLP, with arbitrary precision, by an MILP. To further enhance this MILP, we show how to significantly reduce the number of additional binary variables by leveraging concave areas of the objective function. Extensive experiments demonstrate the efficiency of our approaches.

math.OC↗

arXiv2024

SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning

We focus on offline imitation learning (IL), which aims to mimic an expert's behavior using demonstrations without any interaction with the environment. One of the main challenges in offline IL is the limited support of expert demonstrations, which typically cover only a small fraction of the state-action space. While it may not be feasible to obtain numerous expert demonstrations, it is often possible to gather a larger set of sub-optimal demonstrations. For example, in treatment optimization problems, there are varying levels of doctor treatments available for different chronic conditions. These range from treatment specialists and experienced general practitioners to less experienced general practitioners. Similarly, when robots are trained to imitate humans in routine tasks, they might learn from individuals with different levels of expertise and efficiency. In this paper, we propose an offline IL approach that leverages the larger set of sub-optimal demonstrations while effectively mimicking expert trajectories. Existing offline IL methods based on behavior cloning or distribution matching often face issues such as overfitting to the limited set of expert demonstrations or inadvertently imitating sub-optimal trajectories from the larger dataset. Our approach, which is based on inverse soft-Q learning, learns from both expert and sub-optimal demonstrations. It assigns higher importance (through learned weights) to aligning with expert demonstrations and lower importance to aligning with sub-optimal ones. A key contribution of our approach, called SPRINQL, is transforming the offline IL problem into a convex optimization over the space of Q functions. Through comprehensive experimental evaluations, we demonstrate that the SPRINQL algorithm achieves state-of-the-art (SOTA) performance on offline IL benchmarks. Code is available at https://github.com/hmhuy0/SPRINQL.

cs.LG↗

arXiv2024

UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations

We address the problem of offline learning a policy that avoids undesirable demonstrations. Unlike conventional offline imitation learning approaches that aim to imitate expert or near-optimal demonstrations, our setting involves avoiding undesirable behavior (specified using undesirable demonstrations). To tackle this problem, unlike standard imitation learning where the aim is to minimize the distance between learning policy and expert demonstrations, we formulate the learning task as maximizing a statistical distance, in the space of state-action stationary distributions, between the learning policy and the undesirable policy. This significantly different approach results in a novel training objective that necessitates a new algorithm to address it. Our algorithm, UNIQ, tackles these challenges by building on the inverse Q-learning framework, framing the learning problem as a cooperative (non-adversarial) task. We then demonstrate how to efficiently leverage unlabeled data for practical training. Our method is evaluated on standard benchmark environments, where it consistently outperforms state-of-the-art baselines. The code implementation can be accessed at: https://github.com/hmhuy0/UNIQ.

cs.LG↗

arXiv2024

ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization

Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we introduce a regularizer in the space of stationary distributions to better handle distributional shift. Our algorithm, ComaDICE, offers a principled framework for offline cooperative MARL by incorporating stationary distribution regularization for the global learning policy, complemented by a carefully structured multi-agent value decomposition strategy to facilitate multi-agent training. Through extensive experiments on the multi-agent MuJoCo and StarCraft II benchmarks, we demonstrate that ComaDICE achieves superior performance compared to state-of-the-art offline MARL methods across nearly all tasks.

cs.LG↗

arXiv2024

Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learning

A popular framework for enforcing safe actions in Reinforcement Learning (RL) is Constrained RL, where trajectory based constraints on expected cost (or other cost measures) are employed to enforce safety and more importantly these constraints are enforced while maximizing expected reward. Most recent approaches for solving Constrained RL convert the trajectory based cost constraint into a surrogate problem that can be solved using minor modifications to RL methods. A key drawback with such approaches is an over or underestimation of the cost constraint at each state. Therefore, we provide an approach that does not modify the trajectory based cost constraint and instead imitates ``good'' trajectories and avoids ``bad'' trajectories generated from incrementally improving policies. We employ an oracle that utilizes a reward threshold (which is varied with learning) and the overall cost constraint to label trajectories as ``good'' or ``bad''. A key advantage of our approach is that we are able to work from any starting policy or set of trajectories and improve on it. In an exhaustive set of experiments, we demonstrate that our approach is able to outperform top benchmark approaches for solving Constrained RL problems, with respect to expected cost, CVaR cost, or even unknown cost constraints.

cs.LG↗

arXiv2024

Competitive Facility Location under Cross-Nested Logit Customer Choice Model: Hardness and Exact Approaches

We study the competitive facility location problem, where a firm aims to establish new facilities in a market already occupied by competitors. In this problem, customer behavior is crucial for making optimal location decisions. We explore a general class of customer choice models, known as the cross-nested logit (CNL) model, which is recognized for its flexibility and generality in predicting people's choice behavior. To explore the problem, we first demonstrate that it is NP-hard, even when there is only one customer class. We further show that this hardness result is tight, as the facility location problem under any simpler choice models (such as the logit or nested logit) is polynomial-time solvable when there is one customer class. To tackle the resulting facility location problem, we demonstrate that the objective function under a general cross-nested structure is not concave. Interestingly, we show that by a change of variables, the objective function can be converted to a convex program (i.e., a maximization problem with a concave objective and convex constraints), enabling it to be solved to optimality via an outer-approximation algorithm. Extensive experiments show the efficiency of our approach and provide analyses on the benefits of using the cross-nested model in the facility location context.

math.OC↗

arXiv2024

Outer Approximation and Super-modular Cuts for Constrained Assortment Optimization under Mixed-Logit Model

In this paper, we study the assortment optimization problem under the mixed-logit customer choice model. While assortment optimization has been a major topic in revenue management for decades, the mixed-logit model is considered one of the most general and flexible approaches for modeling and predicting customer purchasing behavior. Existing exact methods have primarily relied on mixed-integer linear programming (MILP) or second-order cone (CONIC) reformulations, which allow for exact problem solving using off-the-shelf solvers. However, these approaches often suffer from weak continuous relaxations and are slow when solving large instances. Our work addresses the problem by focusing on components of the objective function that can be proven to be monotonically super-modular and convex. This allows us to derive valid cuts to outer-approximate the nonlinear objective functions. We then demonstrate that these valid cuts can be incorporated into Cutting Plane or Branch-and-Cut methods to solve the problem exactly. Extensive experiments show that our approaches consistently outperform previous methods in terms of both solution quality and computation time.

math.OC↗