arXiv subjects

Nan Jiang

Publications and source records attributed to Nan Jiang.

At least 199 records · Page 11Linked to original sources

arXiv2022

Offline Reinforcement Learning with Realizability and Single-policy Concentrability

Sample-efficiency guarantees for offline reinforcement learning (RL) often rely on strong assumptions on both the function classes (e.g., Bellman-completeness) and the data coverage (e.g., all-policy concentrability). Despite the recent efforts on relaxing these assumptions, existing works are only able to relax one of the two factors, leaving the strong assumption on the other factor intact. As an important open problem, can we achieve sample-efficient offline RL with weak assumptions on both factors? In this paper we answer the question in the positive. We analyze a simple algorithm based on the primal-dual formulation of MDPs, where the dual variables (discounted occupancy) are modeled using a density-ratio function against offline data. With proper regularization, we show that the algorithm enjoys polynomial sample complexity, under only realizability and single-policy concentrability. We also provide alternative analyses based on different assumptions to shed light on the nature of primal-dual algorithms for offline RL.

cs.LG↗

arXiv2022

Model-free Representation Learning and Exploration in Low-rank MDPs

The low rank MDP has emerged as an important model for studying representation learning and exploration in reinforcement learning. With a known representation, several model-free exploration strategies exist. In contrast, all algorithms for the unknown representation setting are model-based, thereby requiring the ability to model the full dynamics. In this work, we present the first model-free representation learning algorithms for low rank MDPs. The key algorithmic contribution is a new minimax representation learning objective, for which we provide variants with differing tradeoffs in their statistical and computational properties. We interleave this representation learning step with an exploration strategy to cover the state space in a reward-free manner. The resulting algorithms are provably sample efficient and can accommodate general function approximation to scale to complex environments.

cs.LG↗

arXiv2022

A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes

We consider off-policy evaluation (OPE) in Partially Observable Markov Decision Processes (POMDPs), where the evaluation policy depends only on observable variables and the behavior policy depends on unobservable latent variables. Existing works either assume no unmeasured confounders, or focus on settings where both the observation and the state spaces are tabular. In this work, we first propose novel identification methods for OPE in POMDPs with latent confounders, by introducing bridge functions that link the target policy's value and the observed data distribution. We next propose minimax estimation methods for learning these bridge functions, and construct three estimators based on these estimated bridge functions, corresponding to a value function-based estimator, a marginalized importance sampling estimator, and a doubly-robust estimator. Our proposal permits general function approximation and is thus applicable to settings with continuous or large observation/state spaces. The nonasymptotic and asymptotic properties of the proposed estimators are investigated in detail.

cs.LG↗

arXiv2022

Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps

We consider a challenging theoretical problem in offline reinforcement learning (RL): obtaining sample-efficiency guarantees with a dataset lacking sufficient coverage, under only realizability-type assumptions for the function approximators. While the existing theory has addressed learning under realizability and under non-exploratory data separately, no work has been able to address both simultaneously (except for a concurrent work which we compare in detail). Under an additional gap assumption, we provide guarantees to a simple pessimistic algorithm based on a version space formed by marginalized importance sampling (MIS), and the guarantee only requires the data to cover the optimal policy and the function classes to realize the optimal value and density-ratio functions. While similar gap assumptions have been used in other areas of RL theory, our work is the first to identify the utility and the novel mechanism of gap assumptions in offline RL with weak function approximation.

cs.LG↗

arXiv2022

Quantum Support Vector Machine without Iteration

Quantum algorithms can enhance machine learning in different aspects. In 2014, Rebentrost $et~al.$ constructed a least squares quantum support vector machine (LS-QSVM), in which the Swap Test plays a crucial role in realizing the classification. However, as the output states of a previous test cannot be reused for a new test in the Swap Test, the quantum algorithm LS-QSVM has to be repeated in preparing qubits, manipulating operations, and carrying out the measurement. This paper proposes a QSVM based on the generalized quantum amplitude estimation (AE-QSVM) which gets rid of the constraint of repetitive processes and saves the quantum resources. At first, AE-QSVM is trained by using the quantum singular value decomposition. Then, a query sample is classified by using the generalized quantum amplitude estimation in which high accuracy can be achieved by adding auxiliary qubits instead of repeating the algorithm. The complexity of AE-QSVM is reduced to $O(κ^{3}\varepsilon^{-3}(log(mn)+1))$ with an accuracy $\varepsilon$, where $m$ is the number of training vectors, $n$ is the dimension of the feature space, and $κ$ is the condition number. Experiments demonstrate that AE-QSVM is advantageous in terms of training matrix, the number of iterations, space complexity, and time complexity.

quant-ph↗

arXiv2022

Adaptable Semantic Compression and Resource Allocation for Task-Oriented Communications

Task-oriented communication is a new paradigm that aims at providing efficient connectivity for accomplishing intelligent tasks rather than the reception of every transmitted bit. In this paper, a deep learning-based task-oriented communication architecture is proposed where the user extracts, compresses and transmits semantics in an end-to-end (E2E) manner. Furthermore, an approach is proposed to compress the semantics according to their importance relevant to the task, namely, adaptable semantic compression (ASC). Assuming a delay-intolerant system, supporting multiple users indicates a problem that executing with the higher compression ratio requires fewer channel resources but leads to the distortion of semantics, while executing with the lower compression ratio requires more channel resources and thus may lead to a transmission failure due to delay constraint. To solve the problem, both compression ratio and resource allocation are optimized for the task-oriented communication system to maximize the success probability of tasks. Specifically, due to the nonconvexity of the problem, we propose a compression ratio and resource allocation (CRRA) algorithm by separating the problem into two subproblems and solving iteratively to obtain the convergent solution. Furthermore, considering the scenarios where users have various service levels, a compression ratio, resource allocation, and user selection (CRRAUS) algorithm is proposed to deal with the problem. In CRRAUS, users are adaptively selected to complete the corresponding intelligent tasks based on branch and bound method at the expense of higher algorithm complexity compared with CRRA. Simulation results show that the proposed CRRA and CRRAUS algorithms can obtain at least 15% and 10% success gains over baseline algorithms, respectively.

eess.SP↗

arXiv2022

Quantum-Memory-Enhanced Preparation of Nonlocal Graph States

Graph states are an important class of multipartite entangled states. Previous experimental generation of graph states and in particular the Greenberger-Horne-Zeilinger (GHZ) states in linear optics quantum information schemes is subjected to an exponential decay in efficiency versus the system size, which limits its large-scale applications in quantum networks. Here we demonstrate an efficient scheme to prepare graph states with only a polynomial overhead using long-lived atomic quantum memories. We generate atom-photon entangled states in two atomic ensembles asynchronously, retrieve the stored atomic excitations only when both sides succeed, and further project them into a four-photon GHZ state. We measure the fidelity of this GHZ state and further demonstrate its applications in the violation of Bell-type inequalities and in quantum cryptography. Our work demonstrates the prospect of efficient generation of multipartite entangled states in large-scale distributed systems with applications in quantum information processing and metrology.

quant-ph↗

arXiv2022

VRConvMF: Visual Recurrent Convolutional Matrix Factorization for Movie Recommendation

Sparsity of user-to-item rating data becomes one of challenging issues in the recommender systems, which severely deteriorates the recommendation performance. Fortunately, context-aware recommender systems can alleviate the sparsity problem by making use of some auxiliary information, such as the information of both the users and items. In particular, the visual information of items, such as the movie poster, can be considered as the supplement for item description documents, which helps to obtain more item features. In this paper, we focus on movie recommender system and propose a probabilistic matrix factorization based recommendation scheme called visual recurrent convolutional matrix factorization (VRConvMF), which utilizes the textual and multi-level visual features extracted from the descriptive texts and posters respectively. We implement the proposed VRConvMF and conduct extensive experiments on three commonly used real world datasets to validate its effectiveness. The experimental results illustrate that the proposed VRConvMF outperforms the existing schemes.

cs.IR↗

arXiv2022

On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction

In this paper, we study the convergence properties of off-policy policy improvement algorithms with state-action density ratio correction under function approximation setting, where the objective function is formulated as a max-max-min optimization problem. We characterize the bias of the learning objective and present two strategies with finite-time convergence guarantees. In our first strategy, we present algorithm P-SREDA with convergence rate $O(ε^{-3})$, whose dependency on $ε$ is optimal. In our second strategy, we propose a new off-policy actor-critic style algorithm named O-SPIM. We prove that O-SPIM converges to a stationary point with total complexity $O(ε^{-4})$, which matches the convergence rate of some recent actor-critic algorithms in the on-policy setting.

cs.LG↗

arXiv2022

Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning

Recent theoretical work studies sample-efficient reinforcement learning (RL) extensively in two settings: learning interactively in the environment (online RL), or learning from an offline dataset (offline RL). However, existing algorithms and theories for learning near-optimal policies in these two settings are rather different and disconnected. Towards bridging this gap, this paper initiates the theoretical study of policy finetuning, that is, online RL where the learner has additional access to a "reference policy" $μ$ close to the optimal policy $π_\star$ in a certain sense. We consider the policy finetuning problem in episodic Markov Decision Processes (MDPs) with $S$ states, $A$ actions, and horizon length $H$. We first design a sharp offline reduction algorithm -- which simply executes $μ$ and runs offline policy optimization on the collected dataset -- that finds an $\varepsilon$ near-optimal policy within $\widetilde{O}(H^3SC^\star/\varepsilon^2)$ episodes, where $C^\star$ is the single-policy concentrability coefficient between $μ$ and $π_\star$. This offline result is the first that matches the sample complexity lower bound in this setting, and resolves a recent open question in offline RL. We then establish an $Ω(H^3S\min\{C^\star, A\}/\varepsilon^2)$ sample complexity lower bound for any policy finetuning algorithm, including those that can adaptively explore the environment. This implies that -- perhaps surprisingly -- the optimal policy finetuning algorithm is either offline reduction or a purely online RL algorithm that does not use $μ$. Finally, we design a new hybrid offline/online algorithm for policy finetuning that achieves better sample complexity than both vanilla offline reduction and purely online RL algorithms, in a relaxed setting where $μ$ only satisfies concentrability partially up to a certain time step.

cs.LG↗

arXiv2022

Experimental demonstration of memory-enhanced scaling for entanglement connection of quantum repeater segments

The quantum repeater protocol is a promising approach to implement long-distance quantum communication and large-scale quantum networks. A key idea of the quantum repeater protocol is to use long-lived quantum memories to achieve efficient entanglement connection between different repeater segments with a polynomial scaling. Here we report an experiment which realizes efficient connection of two quantum repeater segments via on-demand entanglement swapping by the use of two atomic quantum memories with storage time of tens of milliseconds. With the memory enhancement, scaling-changing acceleration is demonstrated in the rate for a successful entanglement connection. The experimental realization of entanglement connection of two quantum repeater segments with an efficient memory-enhanced scaling demonstrates a key advantage of the quantum repeater protocol, which makes a cornerstone towards future large-scale quantum networks.

quant-ph↗

arXiv2021

Quantifying Spatial Homogeneity of Urban Road Networks via Graph Neural Networks

Quantifying the topological similarities of different parts of urban road networks (URNs) enables us to understand the urban growth patterns. While conventional statistics provide useful information about characteristics of either a single node's direct neighbors or the entire network, such metrics fail to measure the similarities of subnetworks considering local indirect neighborhood relationships. In this study, we propose a graph-based machine-learning method to quantify the spatial homogeneity of subnetworks. We apply the method to 11,790 urban road networks across 30 cities worldwide to measure the spatial homogeneity of road networks within each city and across different cities. We find that intra-city spatial homogeneity is highly associated with socioeconomic statuses such as GDP and population growth. Moreover, inter-city spatial homogeneity obtained by transferring the model across different cities, reveals the inter-city similarity of urban network structures originating in Europe, passed on to cities in the US and Asia. Socioeconomic development and inter-city similarity revealed using our method can be leveraged to understand and transfer insights across cities. It also enables us to address urban policy challenges including network planning in rapidly urbanizing areas and combating regional inequality.

physics.soc-ph↗

arXiv2021

Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

We offer an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to a need for standardized empirical analyses. Our work takes a strong focus on diversity of experimental design to enable stress testing of OPE methods. We provide a comprehensive benchmarking suite to study the interplay of different attributes on method performance. We distill the results into a summarized set of guidelines for OPE in practice. Our software package, the Caltech OPE Benchmarking Suite (COBS), is open-sourced and we invite interested researchers to further contribute to the benchmark.

cs.LG↗

arXiv2021

Improved Worst-Case Regret Bounds for Randomized Least-Squares Value Iteration

This paper studies regret minimization with randomized value functions in reinforcement learning. In tabular finite-horizon Markov Decision Processes, we introduce a clipping variant of one classical Thompson Sampling (TS)-like algorithm, randomized least-squares value iteration (RLSVI). Our $\tilde{\mathrm{O}}(H^2S\sqrt{AT})$ high-probability worst-case regret bound improves the previous sharpest worst-case regret bounds for RLSVI and matches the existing state-of-the-art worst-case TS-based regret bounds.

cs.LG↗

arXiv2021

Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning

How to select between policies and value functions produced by different training algorithms in offline reinforcement learning (RL) -- which is crucial for hyperpa-rameter tuning -- is an important open question. Existing approaches based on off-policy evaluation (OPE) often require additional function approximation and hence hyperparameters, creating a chicken-and-egg situation. In this paper, we design hyperparameter-free algorithms for policy selection based on BVFT [XJ21], a recent theoretical advance in value-function selection, and demonstrate their effectiveness in discrete-action benchmarks such as Atari. To address performance degradation due to poor critics in continuous-action domains, we further combine BVFT with OPE to get the best of both worlds, and obtain a hyperparameter-tuning method for Q-function based OPE with theoretical guarantees as a side product.

cs.LG↗

arXiv2021

Multicell Atomic Quantum Memory as a Hardware-Efficient Quantum Repeater Node

For scalable quantum communication and networks, a key step is to realize a quantum repeater node that can efficiently connect different segments of atom-photon entanglement using quantum memories. We report a compact and hardware-efficient realization of a quantum repeater node using a single atomic ensemble for multicell quantum memories. Millisecond lifetime is achieved for individual memory cells after suppressing the magnetic-field-induced inhomogeneous broadening and the atomic-motion-induced spin-wave dephasing. Based on these long-lived multicell memory cells, we achieve heralded asynchronous entanglement generation in two quantum repeater segments one after another and then an on-demand entanglement connection of these two repeater segments. As another application of the multicell atomic quantum memory, we further demonstrate storage and on-demand retrieval of heralded atomic spin-wave qubits by implementing a random access quantum memory with individual addressing capacity. This work provides a promising constituent for efficient realization of quantum repeaters for large-scale quantum networks.

quant-ph↗

arXiv2021

ALSO-X and ALSO-X+: Better Convex Approximations for Chance Constrained Programs

In a chance constrained program (CCP), the decision-makers aim to seek the best decision whose probability of violating the uncertainty constraints is within the prespecified risk level. As a CCP is often nonconvex and is difficult to solve to optimality, much effort has been devoted to developing convex inner approximations for a CCP, among which the conditional value-at-risk (CVaR) has been known to be the best for more than a decade. This paper studies and generalizes the ALSO-X, originally proposed by Ahmed, Luedtke, SOng, and Xie (2017), for solving a CCP. We first show that the ALSO-X resembles a bilevel optimization, where the upper-level problem is to find the best objective function value and enforce the feasibility of a CCP for a given decision from the lower-level problem, and the lower-level problem is to minimize the expectation of constraint violations subject to the upper bound of the objective function value provided by the upper-level problem. This interpretation motivates us to prove that when uncertain constraints are convex in the decision variables, ALSO-X always outperforms the CVaR approximation. We further show (i) sufficient conditions under which ALSO-X can recover an optimal solution to a CCP; (ii) an equivalent bilinear programming formulation of a CCP, inspiring us to enhance ALSO-X with a convergent alternating minimization method (ALSO-X+); (iii) extensions of ALSO-X and ALSO-X+ to solve distributionally robust chance constrained programs (DRCCPs) under $\infty$-Wasserstein ambiguity set. Our numerical study demonstrates the effectiveness of the proposed methods.

math.OC↗

arXiv2021

A Fast Randomized Algorithm for Massive Text Normalization

Many popular machine learning techniques in natural language processing and data mining rely heavily on high-quality text sources. However real-world text datasets contain a significant amount of spelling errors and improperly punctuated variants where the performance of these models would quickly deteriorate. Moreover, real-world, web-scale datasets contain hundreds of millions or even billions of lines of text, where the existing text cleaning tools are prohibitively expensive to execute over and may require an overhead to learn the corrections. In this paper, we present FLAN, a scalable randomized algorithm to clean and canonicalize massive text data. Our algorithm relies on the Jaccard similarity between words to suggest correction results. We efficiently handle the pairwise word-to-word comparisons via Locality Sensitive Hashing (LSH). We also propose a novel stabilization process to address the issue of hash collisions between dissimilar words, which is a consequence of the randomized nature of LSH and is exacerbated by the massive scale of real-world datasets. Compared with existing approaches, our method is more efficient, both asymptotically and in empirical evaluations, and does not rely on additional features, such as lexical/phonetic similarity or word embedding features. In addition, FLAN does not require any annotated data or supervised learning. We further theoretically show the robustness of our algorithm with upper bounds on the false positive and false negative rates of corrections. Our experimental results on real-world datasets demonstrate the efficiency and efficacy of FLAN.

cs.AI↗