arXiv subjects

Mateusz Ostaszewski

Publications and source records attributed to Mateusz Ostaszewski.

30 records · Page 2Linked to original sources

arXiv2023

The Effectiveness of World Models for Continual Reinforcement Learning

World models power some of the most efficient reinforcement learning algorithms. In this work, we showcase that they can be harnessed for continual learning - a situation when the agent faces changing environments. World models typically employ a replay buffer for training, which can be naturally extended to continual learning. We systematically study how different selective experience replay methods affect performance, forgetting, and transfer. We also provide recommendations regarding various modeling options for using world models. The best set of choices is called Continual-Dreamer, it is task-agnostic and utilizes the world model for continual exploration. Continual-Dreamer is sample efficient and outperforms state-of-the-art task-agnostic continual reinforcement learning methods on Minigrid and Minihack benchmarks.

cs.LG↗

arXiv2022

Emergency action termination for immediate reaction in hierarchical reinforcement learning

Hierarchical decomposition of control is unavoidable in large dynamical systems. In reinforcement learning (RL), it is usually solved with subgoals defined at higher policy levels and achieved at lower policy levels. Reaching these goals can take a substantial amount of time, during which it is not verified whether they are still worth pursuing. However, due to the randomness of the environment, these goals may become obsolete. In this paper, we address this gap in the state-of-the-art approaches and propose a method in which the validity of higher-level actions (thus lower-level goals) is constantly verified at the higher level. If the actions, i.e. lower level goals, become inadequate, they are replaced by more appropriate ones. This way we combine the advantages of hierarchical RL, which is fast training, and flat RL, which is immediate reactivity. We study our approach experimentally on seven benchmark environments.

cs.LG↗

arXiv2022

Reinforcement learning with experience replay and adaptation of action dispersion

Effective reinforcement learning requires a proper balance of exploration and exploitation defined by the dispersion of action distribution. However, this balance depends on the task, the current stage of the learning process, and the current environment state. Existing methods that designate the action distribution dispersion require problem-dependent hyperparameters. In this paper, we propose to automatically designate the action distribution dispersion using the following principle: This distribution should have sufficient dispersion to enable the evaluation of future policies. To that end, the dispersion should be tuned to assure a sufficiently high probability (densities) of the actions in the replay buffer and the modes of the distributions that generated them, yet this dispersion should not be higher. This way, a policy can be effectively evaluated based on the actions in the buffer, but exploratory randomness in actions decreases when this policy converges. The above principle is verified here on challenging benchmarks Ant, HalfCheetah, Hopper, and Walker2D, with good results. Our method makes the action standard deviations converge to values similar to those resulting from trial-and-error optimization.

cs.LG↗

arXiv2021

Reinforcement learning for optimization of variational quantum circuit architectures

The study of Variational Quantum Eigensolvers (VQEs) has been in the spotlight in recent times as they may lead to real-world applications of near-term quantum devices. However, their performance depends on the structure of the used variational ansatz, which requires balancing the depth and expressivity of the corresponding circuit. In recent years, various methods for VQE structure optimization have been introduced but the capacities of machine learning to aid with this problem has not yet been fully investigated. In this work, we propose a reinforcement learning algorithm that autonomously explores the space of possible ans{ä}tze, identifying economic circuits which still yield accurate ground energy estimates. The algorithm is intrinsically motivated, and it incrementally improves the accuracy of the result while minimizing the circuit depth. We showcase the performance of our algorithm on the problem of estimating the ground-state energy of lithium hydride (LiH). In this well-known benchmark problem, we achieve chemical accuracy, as well as state-of-the-art results in terms of circuit depth.

quant-ph↗

arXiv2021

Structure optimization for parameterized quantum circuits

We propose an efficient method for simultaneously optimizing both the structure and parameter values of quantum circuits with only a small computational overhead. Shallow circuits that use structure optimization perform significantly better than circuits that use parameter updates alone, making this method particularly suitable for noisy intermediate-scale quantum computers. We demonstrate the method for optimizing a variational quantum eigensolver for finding the ground states of Lithium Hydride and the Heisenberg model in simulation, and for finding the ground state of Hydrogen gas on the IBM Melbourne quantum computer.

quant-ph↗

arXiv2020

Effective training of deep convolutional neural networks for hyperspectral image classification through artificial labeling

Hyperspectral imaging is a rich source of data, allowing for multitude of effective applications. However, such imaging remains challenging because of large data dimension and, typically, small pool of available training examples. While deep learning approaches have been shown to be successful in providing effective classification solutions, especially for high dimensional problems, unfortunately they work best with a lot of labelled examples available. To alleviate the second requirement for a particular dataset the transfer learning approach can be used: first the network is pre-trained on some dataset with large amount of training labels available, then the actual dataset is used to fine-tune the network. This strategy is not straightforward to apply with hyperspectral images, as it is often the case that only one particular image of some type or characteristic is available. In this paper, we propose and investigate a simple and effective strategy of transfer learning that uses unsupervised pre-training step without label information. This approach can be applied to many of the hyperspectral classification problems. Performed experiments show that it is very effective at improving the classification accuracy without being restricted to a particular image type or neural network architecture. The experiments were carried out on several deep neural network architectures and various sizes of labeled training sets. The greatest improvement in overall accuracy on the Indian Pines and Pavia University datasets is over 21 and 13 percentage points, respectively. An additional advantage of the proposed approach is the unsupervised nature of the pre-training step, which can be done immediately after image acquisition, without the need of the potentially costly expert's time.

cs.NE↗

arXiv2019

An initialization strategy for addressing barren plateaus in parametrized quantum circuits

Parametrized quantum circuits initialized with random initial parameter values are characterized by barren plateaus where the gradient becomes exponentially small in the number of qubits. In this technical note we theoretically motivate and empirically validate an initialization strategy which can resolve the barren plateau problem for practical applications. The technique involves randomly selecting some of the initial parameter values, then choosing the remaining values so that the circuit is a sequence of shallow blocks that each evaluates to the identity. This initialization limits the effective depth of the circuits used to calculate the first parameter update so that they cannot be stuck in a barren plateau at the start of training. In turn, this makes some of the most compact ansätze usable in practice, which was not possible before even for rather basic problems. We show empirically that variational quantum eigensolvers and quantum neural networks initialized using this strategy can be trained using a gradient based method.

quant-ph↗

arXiv2018

QSWalk.jl: Julia package for quantum stochastic walks analysis

The presented paper describes QSWalk.jl package for Julia programming language, developed for the purpose of simulating the evolution of open quantum systems. The package enables the study of quantum procedures developed using stochastic quantum walks on arbitrary directed graphs. We provide a detailed description of the implemented functions, along with a number of usage examples. The package is compared with the existing software offering a similar functionality.

quant-ph↗

arXiv2017

Limit properties of global interaction stochastic quantum walks on directed graphs

The main results of our work is determining the differences between limiting properties in various models of quantum stochastic walks. In particular, we prove that in the case of strongly connected and a class of weakly connected directed graphs, local environment interaction evolution is relaxing, and in the case of undirected graphs, global environment interaction evolution is convergent. For other classes of directed graphs we show, that the character of connectivity large influence on the limiting properties. We also study the limiting properties for the non-moralizing global interaction case. We demonstrate that the digraph observance is recovered in this case.

quant-ph↗

arXiv2017

Properties of quantum stochastic walks from the asymptotic scaling exponent

This work focuses on the study of quantum stochastic walks, which are a generalization of coherent, i. e. unitary quantum walks. Our main goal is to present a measure of a coherence of the walk. To this end, we utilize the asymptotic scaling exponent of the second moment of the walk i. e. of the mean squared distance covered by a walk. As the quantum stochastic walk model encompasses both classical random walks and quantum walks, we are interested how the continuous change from one regime to the other influences the asymptotic scaling exponent. Moreover this model allows for behavior which is not found in any of the previously mentioned model -- the model with global dissipation. We derive the probability distribution for the walker, and determine the asymptotic scaling exponent analytically, showing that ballistic regime of the walk is maintained even at large dissipation strength.

quant-ph↗

arXiv2017

Lively quantum walks on cycles

We introduce a family of quantum walks on cycles parametrized by their liveliness, defined by the ability to execute a long-range move. We investigate the behaviour of the probability distribution and time-averaged probability distribution. We show that the liveliness parameter, controlling the magnitude of the additional long-range move, has a direct impact on the periodicity of the limiting distribution. We also show that the introduced model provides a method for network exploration which is robust against trapping.

quant-ph↗

arXiv2015

Quantum image classification using principal component analysis

We present a novel quantum algorithm for classification of images. The algorithm is constructed using principal component analysis and von Neuman quantum measurements. In order to apply the algorithm we present a new quantum representation of grayscale images.

quant-ph↗