arXiv subjects

Nan Jiang

Publications and source records attributed to Nan Jiang.

At least 235 records · Page 13Linked to original sources

arXiv2020

Minimax Weight and Q-Function Learning for Off-Policy Evaluation

We provide theoretical investigations into off-policy evaluation in reinforcement learning using function approximators for (marginalized) importance weights and value functions. Our contributions include: (1) A new estimator, MWL, that directly estimates importance ratios over the state-action distributions, removing the reliance on knowledge of the behavior policy as in prior work (Liu et al., 2018). (2) Another new estimator, MQL, obtained by swapping the roles of importance weights and value-functions in MWL. MQL has an intuitive interpretation of minimizing average Bellman errors and can be combined with MWL in a doubly robust manner. (3) Several additional results that offer further insights into these methods, including the sample complexity analyses of MWL and MQL, their asymptotic optimality in the tabular setting, how the learned importance weights depend the choice of the discriminator class, and how our methods provide a unified view of some old and new algorithms in RL.

cs.LG↗

arXiv2020

The 1st Tiny Object Detection Challenge:Methods and Results

The 1st Tiny Object Detection (TOD) Challenge aims to encourage research in developing novel and accurate methods for tiny object detection in images which have wide views, with a current focus on tiny person detection. The TinyPerson dataset was used for the TOD Challenge and is publicly released. It has 1610 images and 72651 box-levelannotations. Around 36 participating teams from the globe competed inthe 1st TOD Challenge. In this paper, we provide a brief summary of the1st TOD Challenge including brief introductions to the top three methods.The submission leaderboard will be reopened for researchers that areinterested in the TOD challenge. The benchmark dataset and other information can be found at: https://github.com/ucas-vg/TinyBenchmark.

cs.CV↗

arXiv2020

Analytic I-Love-C relations for realistic neutron stars

Recent observations of neutron stars (NS) with radio, X-rays and gravitational waves have begun to constrain the equation of state (EoS) for nuclear matter beyond the nuclear saturation density. To one's surprise, there exist approximate universal relations connecting certain bulk properties of NSs that are insensitive to the underlying EoS and having important applications on probing fundamental physics including nuclear and gravitational physics. To date, analytic works on universal relations for realistic NSs are lacking, which may lead to a better understanding of the origin of the universality. Here, we focus on the universal relations between the compactness(C), moment of inertia (I), and tidal deformability(related to the Love number), and derive analytic, approximate I-Love-C relations. To achieve this, we construct slowly-rotating/ tidally-deformed NS solutions analytically starting from an extended Tolman VII model that accurately describes non-rotating realistic NSs, allowing us to extract the moment of inertia and the tidal deformability on top of the compactness. We solve the field equations analytically by expanding them about the Newtonian limit and keeping up to 6th order in the stellar compactness. Based on these analytic solutions, we can mathematically demonstrate the O(10%) EoS variation in the I-C and Love-C relations and the O(1%) variation in the I-Love relation that have previously been found numerically. Our new analytic relations agree more accurately with numerical results for realistic NSs(especially the I-C and Love-C ones) than the analytic relations for constant density stars derived in previous work. Based on these analytic findings, we attribute a possible origin of the universality for the I-C and Love-C relations to the fact that the energy density of realistic NSs can be approximated as a quadratic function, as is the case for the Tolman VII solution.

gr-qc↗

arXiv2020

Analysis of Random Access in NB-IoT Networks with Three Coverage Enhancement Groups: A Stochastic Geometry Approach

NarrowBand-Internet of Things (NB-IoT) is a new 3GPP radio access technology designed to provide better coverage for Low Power Wide Area (LPWA) networks. To provide reliable connections with extended coverage, a repetition transmission scheme and up to three Coverage Enhancement (CE) groups are introduced into NB-IoT during both Random Access CHannel (RACH) procedure and data transmission procedure, where each CE group is configured with different repetition values and transmission resources. To characterize the RACH performance of the NB-IoT network with three CE groups, this paper develops a novel traffic-aware spatio-temporal model to analyze the RACH success probability, where both the preamble transmission outage and the collision events of each CE group jointly determine the traffic evolution and the RACH success probability. Based on this analytical model, we derive the analytical expression for the RACH success probability of a randomly chosen IoT device in each CE group over multiple time slots with different RACH schemes, including baseline, back-off (BO), access class barring (ACB), and hybrid ACB and BO schemes (ACB&BO). Our results have shown that the RACH success probabilities of the devices in three CE groups outperform that of a single CE group network but not for all the groups, which is affected by the choice of the categorizing parameters.This mathematical model and analytical framework can be applied to evaluate the performance of multiple group users of other networks with spatial separations.

eess.SP↗

arXiv2020

Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison

We prove performance guarantees of two algorithms for approximating $Q^\star$ in batch reinforcement learning. Compared to classical iterative methods such as Fitted Q-Iteration---whose performance loss incurs quadratic dependence on horizon---these methods estimate (some forms of) the Bellman error and enjoy linear-in-horizon error propagation, a property established for the first time for algorithms that rely solely on batch data and output stationary policies. One of the algorithms uses a novel and explicit importance-weighting correction to overcome the infamous "double sampling" difficulty in Bellman error estimation, and does not use any squared losses. Our analyses reveal its distinct characteristics and potential advantages compared to classical algorithms.

cs.LG↗

arXiv2020

From Importance Sampling to Doubly Robust Policy Gradient

We show that on-policy policy gradient (PG) and its variance reduction variants can be derived by taking finite difference of function evaluations supplied by estimators from the importance sampling (IS) family for off-policy evaluation (OPE). Starting from the doubly robust (DR) estimator (Jiang & Li, 2016), we provide a simple derivation of a very general and flexible form of PG, which subsumes the state-of-the-art variance reduction technique (Cheng et al., 2019) as its special case and immediately hints at further variance reduction opportunities overlooked by existing literature. We analyze the variance of the new DR-PG estimator, compare it to existing methods as well as the Cramer-Rao lower bound of policy gradient, and empirically show its effectiveness.

cs.LG↗

arXiv2020

On Value Functions and the Agent-Environment Boundary

When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely defined as they depend on where we draw this agent-environment boundary, causing problems in theoretical analyses that provide optimality guarantees. We address this issue via a simple and novel boundary-invariant analysis of Fitted Q-Iteration, a representative RL algorithm, where the assumptions and the guarantees are invariant to the choice of boundary. We also discuss closely related issues on state resetting and Monte-Carlo Tree Search, deterministic vs stochastic systems, imitation learning, and the verifiability of theoretical assumptions from data.

cs.LG↗

arXiv2020

A Question Type Driven and Copy Loss Enhanced Frameworkfor Answer-Agnostic Neural Question Generation

The answer-agnostic question generation is a significant and challenging task, which aims to automatically generate questions for a given sentence but without an answer. In this paper, we propose two new strategies to deal with this task: question type prediction and copy loss mechanism. The question type module is to predict the types of questions that should be asked, which allows our model to generate multiple types of questions for the same source sentence. The new copy loss enhances the original copy mechanism to make sure that every important word in the source sentence has been copied when generating questions. Our integrated model outperforms the state-of-the-art approach in answer-agnostic question generation, achieving a BLEU-4 score of 13.9 on SQuAD. Human evaluation further validates the high quality of our generated questions. We will make our code public available for further research.

cs.CL↗

arXiv2020

A Decoupled Learning Strategy for Massive Access Optimization in Cellular IoT Networks

Cellular-based networks are expected to offer connectivity for massive Internet of Things (mIoT) systems. However, their Random Access CHannel (RACH) procedure suffers from unreliability, due to the collision from the simultaneous massive access. Despite that this collision problem has been treated in existing RACH schemes, these schemes usually organize IoT devices' transmission and re-transmission along with fixed parameters, thus can hardly adapt to time-varying traffic patterns. Without adaptation, the RACH procedure easily suffers from high access delay, high energy consumption, or even access unavailability. With the goal of improving the RACH procedure, this paper targets to optimize the RACH procedure in real-time by maximizing a long-term hybrid multi-objective function, which consists of the number of access success devices, the average energy consumption, and the average access delay. To do so, we first optimize the long-term objective in the number of access success devices by using Deep Reinforcement Learning (DRL) algorithms for different RACH schemes, including Access Class Barring (ACB), Back-Off (BO), and Distributed Queuing (DQ). The converging capability and efficiency of different DRL algorithms including Policy Gradient (PG), Actor-Critic (AC), Deep Q-Network (DQN), and Deep Deterministic Policy Gradients (DDPG) are compared. Inspired by the results from this comparison, a decoupled learning strategy is developed to jointly and dynamically adapt the access control factors of those three access schemes. This decoupled strategy first leverage a Recurrent Neural Network (RNN) model to predict the real-time traffic values of the network environment, and then uses multiple DRL agents to cooperatively configure parameters of each RACH scheme.

cs.NI↗

arXiv2020

Traffic Prediction and Random Access Control Optimization: Learning and Non-learning based Approaches

Random access schemes in modern wireless communications are generally based on the framed-ALOHA (f-ALOHA), which can be optimized by flexibly organizing devices' transmission and re-transmission. However, this optimization is generally intractable due to the lack of information about complex traffic generation statistics and the occurrence of the random collision. In this article, we first summarize the general structure of access control optimization for different random access schemes, and then review the existing access control optimization based on Machine Learning (ML) and non-ML techniques. We demonstrate that the ML-based methods can better optimize the access control problem compared with non-ML based methods, due to their capability in solving high complexity long-term optimization problem and learning experiential knowledge from reality. To further improve the random access performance, we propose two-step learning optimizers for access control optimization, which individually execute the traffic prediction and the access control configuration. In detail, our traffic prediction method relies on online supervised learning adopting Recurrent Neural Networks (RNNs) that can accurately capture traffic statistics over consecutive frames, and the access control configuration can use either a non-ML based controller or a cooperatively trained Deep Reinforcement Learning (DRL) based controller depending on the complexity of different random access schemes. Numerical results show that the proposed two-step cooperative learning optimizer considerably outperforms the conventional Deep Q-Network (DQN) in terms of higher training efficiency and better access performance.

cs.NI↗

arXiv2020

Provably Efficient Q-Learning with Low Switching Cost

We take initial steps in studying PAC-MDP algorithms with limited adaptivity, that is, algorithms that change its exploration policy as infrequently as possible during regret minimization. This is motivated by the difficulty of running fully adaptive algorithms in real-world applications (such as medical domains), and we propose to quantify adaptivity using the notion of local switching cost. Our main contribution, Q-Learning with UCB2 exploration, is a model-free algorithm for H-step episodic MDP that achieves sublinear regret whose local switching cost in K episodes is $O(H^3SA\log K)$, and we provide a lower bound of $Ω(HSA)$ on the local switching cost for any no-regret algorithm. Our algorithm can be naturally adapted to the concurrent setting, which yields nontrivial results that improve upon prior work in certain aspects.

cs.LG↗

arXiv2020

RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning

This paper presents a deep reinforcement learning algorithm for online accompaniment generation, with potential for real-time interactive human-machine duet improvisation. Different from offline music generation and harmonization, online music accompaniment requires the algorithm to respond to human input and generate the machine counterpart in a sequential order. We cast this as a reinforcement learning problem, where the generation agent learns a policy to generate a musical note (action) based on previously generated context (state). The key of this algorithm is the well-functioning reward model. Instead of defining it using music composition rules, we learn this model from monophonic and polyphonic training data. This model considers the compatibility of the machine-generated note with both the machine-generated context and the human-generated context. Experiments show that this algorithm is able to respond to the human part and generate a melodic, harmonic and diverse machine part. Subjective evaluations on preferences show that the proposed algorithm generates music pieces of higher quality than the baseline method.

cs.LG↗

arXiv2019

Scale Match for Tiny Person Detection

Visual object detection has achieved unprecedented ad-vance with the rise of deep convolutional neural networks.However, detecting tiny objects (for example tiny per-sons less than 20 pixels) in large-scale images remainsnot well investigated. The extremely small objects raisea grand challenge about feature representation while themassive and complex backgrounds aggregate the risk offalse alarms. In this paper, we introduce a new benchmark,referred to as TinyPerson, opening up a promising directionfor tiny object detection in a long distance and with mas-sive backgrounds. We experimentally find that the scale mis-match between the dataset for network pre-training and thedataset for detector learning could deteriorate the featurerepresentation and the detectors. Accordingly, we proposea simple yet effective Scale Match approach to align theobject scales between the two datasets for favorable tiny-object representation. Experiments show the significantperformance gain of our proposed approach over state-of-the-art detectors, and the challenging aspects of TinyPersonrelated to real-world scenarios. The TinyPerson benchmarkand the code for our approach will be publicly available(https://github.com/ucas-vg/TinyBenchmark).(Attention: evaluation rules of AP have updated in benchmark after this paper accepted, So this paper use old rules. we will keep old rules of AP in benchmark, but we recommand the new and we will use the new in latter research.)

cs.CV↗

arXiv2019

Why Do Neural Response Generation Models Prefer Universal Replies?

Recent advances in sequence-to-sequence learning reveal a purely data-driven approach to the response generation task. Despite its diverse applications, existing neural models are prone to producing short and generic replies, making it infeasible to tackle open-domain challenges. In this research, we analyze this critical issue in light of the model's optimization goal and the specific characteristics of the human-to-human dialog corpus. By decomposing the black box into parts, a detailed analysis of the probability limit was conducted to reveal the reason behind these universal replies. Based on these analyses, we propose a max-margin ranking regularization term to avoid the models leaning to these replies. Finally, empirical experiments on case studies and benchmarks with several metrics validate this approach.

cs.CL↗

arXiv2019

High dimensional entanglement between a photon and a multiplexed atomic quantum memory

Multiplexed quantum memories and high-dimensional entanglement can improve the performance of quantum repeaters by promoting the entanglement generation rate and the quantum communication channel capacity. Here, we experimentally generate a high-dimensional entangled state between a photon and a collective spin wave excitation stored in the multiplexed atomic quantum memory. We verify the entanglement dimension by the quantum witness and the entanglement of formation. Then we use the high-dimensional entangled state to test the violation of the Bell-type inequality. Our work provides an effective method to generate multidimensional entanglement between the flying photonic pulses and the atomic quantum interface.

quant-ph↗

arXiv2019

Torsional refrigeration by twisted, coiled, and supercoiled fibers

Higher efficiency, lower cost refrigeration is needed for both large and small scale cooling. Refrigerators using entropy changes during cycles of stretching or hydrostatically compression of a solid are possible alternatives to the vapor-compression fridges found in homes. We show that high cooling results from twist changes for twisted, coiled, or supercoiled fibers, including those of natural rubber, NiTi, and polyethylene fishing line. By using opposite chiralities of twist and coiling, supercoiled natural rubber fibers and coiled fishing line fibers result that cool when stretched. A demonstrated twist-based device for cooling flowing water provides a high cooling energy and device efficiency. Theory describes the axial and spring index dependencies of twist-enhanced cooling and its origin in a phase transformation for polyethylene fibers.

physics.app-ph↗

arXiv2019

Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensembles

Reinforcement learning (RL) methods have been shown to be capable of learning intelligent behavior in rich domains. However, this has largely been done in simulated domains without adequate focus on the process of building the simulator. In this paper, we consider a setting where we have access to an ensemble of pre-trained and possibly inaccurate simulators (models). We approximate the real environment using a state-dependent linear combination of the ensemble, where the coefficients are determined by the given state features and some unknown parameters. Our proposed algorithm provably learns a near-optimal policy with a sample complexity polynomial in the number of unknown parameters, and incurs no dependence on the size of the state (or action) space. As an extension, we also consider the more challenging problem of model selection, where the state features are unknown and can be chosen from a large candidate set. We provide exponential lower bounds that illustrate the fundamental hardness of this problem, and develop a provably efficient algorithm under additional natural assumptions.

cs.LG↗

arXiv2019

Quantum Communication between Multiplexed Atomic Quantum Memories

The use of multiplexed atomic quantum memories (MAQM) can significantly enhance the efficiency to establish entanglement in a quantum network. In the previous experiments, individual elements of a quantum network, such as the generation, storage and transmission of quantum entanglement have been demonstrated separately. Here we report an experiment to show the compatibility of these basic operations. Specifically, we generate photon-atom entanglement in a $6\times 5$ MAQM, convert the spin wave to time-bin photonic excitation after a controllable storage time, and then store and retrieve the photon in a second MAQM for another controllable storage time. The preservation of quantum information in this process is verified by measuring the state fidelity. We also show that our scheme supports quantum systems with higher dimension than a qubit.

quant-ph↗