arXiv subjects

Archie C. Chapman

Publications and source records attributed to Archie C. Chapman.

17 recordsLinked to original sources

arXiv2025

Training Directional Locomotion for Quadrupedal Low-Cost Robotic Systems via Deep Reinforcement Learning

In this work we present Deep Reinforcement Learning (DRL) training of directional locomotion for low-cost quadrupedal robots in the real world. In particular, we exploit randomization of heading that the robot must follow to foster exploration of action-state transitions most useful for learning both forward locomotion as well as course adjustments. Changing the heading in episode resets to current yaw plus a random value drawn from a normal distribution yields policies able to follow complex trajectories involving frequent turns in both directions as well as long straight-line stretches. By repeatedly changing the heading, this method keeps the robot moving within the training platform and thus reduces human involvement and need for manual resets during the training. Real world experiments on a custom-built, low-cost quadruped demonstrate the efficacy of our method with the robot successfully navigating all validation tests. When trained with other approaches, the robot only succeeds in forward locomotion test and fails when turning is required.

cs.RO↗

arXiv2025

Low-cost Real-world Implementation of the Swing-up Pendulum for Deep Reinforcement Learning Experiments

Deep reinforcement learning (DRL) has had success in virtual and simulated domains, but due to key differences between simulated and real-world environments, DRL-trained policies have had limited success in real-world applications. To assist researchers to bridge the \textit{sim-to-real gap}, in this paper, we describe a low-cost physical inverted pendulum apparatus and software environment for exploring sim-to-real DRL methods. In particular, the design of our apparatus enables detailed examination of the delays that arise in physical systems when sensing, communicating, learning, inferring and actuating. Moreover, we wish to improve access to educational systems, so our apparatus uses readily available materials and parts to reduce cost and logistical barriers. Our design shows how commercial, off-the-shelf electronics and electromechanical and sensor systems, combined with common metal extrusions, dowel and 3D printed couplings provide a pathway for affordable physical DRL apparatus. The physical apparatus is complemented with a simulated environment implemented using a high-fidelity physics engine and OpenAI Gym interface.

cs.LG↗

arXiv2022

Fair Coordination of Distributed Energy Resources with Volt-Var Control and PV Curtailment

This paper presents a novel distributed optimal power flow (DOPF) method for fair distributed energy resource (DER) coordination in the context of mandated rooftop PV inverter control modes. In practice, inverter reactive power control is increasingly required by grid connection codes, which often unfairly curtail PV generation of prosumers towards the end of low-voltage feeders. Similarly, optimization-based DER coordination methods that aim solely for technically-efficient DER coordination do not consider the distribution of PV curtailment across customers. To address these concerns, we develop a tractable multi-objective DOPF method for optimal DER coordination that (i) curtails PV generation fairly across prosumers, and (ii) incorporates a standard piecewise-linear volt-var control reactive power control function without using integer variables. Three equity principles representing different interpretations of fairness are implemented in our coordination method; namely, egalitarian, proportional and uniform dynamic PV curtailment redistribution. The performance of our approach is demonstrated on low-voltage distribution feeders of different sizes (5, 10, 25, 50 and 100 prosumers) using two network topologies: line topology without lateral spurs and tree topology with lateral spurs. Each network considers three levels of PV penetration, giving 30 test systems in total. The results demonstrate the effectiveness of the proposed DOPF method for fair DER coordination: PV curtailment is equitably distributed among prosumers with a computational burden on par with conventional DOPF approaches. Moreover, different fairness methods result in different patterns of curtailment, which a regulator may choose between.

cs.CE↗

arXiv2022

Practical Considerations of DER Coordination with Distributed Optimal Power Flow

The coordination of prosumer-owned, behind-the-meter distributed energy resources (DER) can be achieved using a multiperiod, distributed optimal power flow (DOPF), which satisfies network constraints and preserves the privacy of prosumers. To solve the problem in a distributed fashion, it is decomposed and solved using the alternating direction method of multipliers (ADMM), which may require many iterations between prosumers and the central entity (i.e., an aggregator). Furthermore, the computational burden is shared among the agents with different processing capacities. Therefore, computational constraints and communication requirements may make the DOPF infeasible or impractical. In this paper, part of the DOPF (some of the prosumer subproblems) is executed on a Raspberry Pi-based hardware prototype, which emulates a low processing power, edge computing device. Four important aspects are analyzed using test cases of different complexities. The first is the computation cost of executing the subproblems in the edge computing device. The second is the algorithm operation on congested electrical networks, which impacts the convergence speed of DOPF solutions. Third, the precision of the computed solution, including the trade-off between solution quality and the number of iterations, is examined. Fourth, the communication requirements for implementation across different communication networks are investigated. The above metrics are analyzed in four scenarios involving 26-bus and 51-bus networks.

cs.CE↗

arXiv2021

Can phase change materials in building insulation improve self-consumption of residential rooftop solar? An Australian case study

This work investigates the extent to which phase change material (PCM) in the building's envelope can be used as an alternative to battery storage systems to increase self-consumption of rooftop solar photovoltaic (PV) generation. In particular, we explore the electricity cost-savings and increase in PV self-consumption that can be achieved by using PCMs and the operation of the heating, ventilation, and air conditioning (HVAC) system optimised by a home energy management system (HEMS). In more detail, we consider a HEMS with an HVAC system, rooftop PV, and a PCM layer integrated into the building envelope. The objective of the HEMS optimisation is to minimise electricity costs while maximising PV self-consumption and maintaining the indoor building temperature in a preferred comfort range. Solving this problem is challenging due to PCM's nonlinear characteristics, and using methods that can deal with the resulting non-convexity of the optimisation problem, like dynamic programming is computationally expensive. Therefore, we use multi-timescale approximate dynamic programming (MADP) that we developed in our earlier work to explore a number of Australian PCM scenarios. Specifically, we analyse a large number of residential buildings across five Australian capital cities. We find that using PCM can reduce annual electricity costs by between 10.6% in Brisbane and 19% in Adelaide. However, somewhat surprisingly, using PCM reduces PV self-consumption by between 1.5% in Brisbane and 2.7% in Perth.

eess.SY↗

arXiv2019

Macro-action Multi-time scale Dynamic Programming for Energy Management in Buildings with Phase Change Materials

This paper focuses on energy management in buildings with phase change material (PCM), which is primarily used to improve thermal performance, but can also serve as an energy storage system. In this setting, optimal scheduling of an HVAC system is challenging because of the nonlinear and non-convex characteristics of the PCM, which makes solving the corresponding optimization problem using conventional optimization techniques impractical. Instead, we use dynamic programming (DP) to deal with the nonlinear nature of the PCM. To overcome DP's curse of dimensionality, this paper proposes a novel methodology to reduce the computational burden, while maintaining the quality of the solution. Specifically, the method incorporates approaches from sequential decision making in artificial intelligence, including macro actions and multi-time scale Markov decision processes, coupled with an underlying state-space approximation to reduce the state-space and action-space size. The performance of the method is demonstrated on an energy management problem for a typical residential building located in Sydney, Australia. The results demonstrate that the proposed method performs well with a computational speed-up of up to 12,900 times compared to the direct application of DP.

cs.LG↗

arXiv2019

A Turvey-Shapley Value Method for Distribution Network Cost Allocation

This paper proposes a novel cost-reflective and computationally efficient method for allocating distribution network costs to residential customers. First, the method estimates the growth in peak demand with a 50% probability of exceedance (50POE) and the associated network augmentation costs using a probabilistic long-run marginal cost computation based on the Turvey perturbation method. Second, it allocates these costs to customers on a cost-causal basis using the Shapley value solution concept. To overcome the intractability of the exact Shapley value computation for real-world applications, we implement a fast, scalable and efficient clustering technique based on customers' peak demand contribution, which drastically reduces the Shapley value computation time. Using customer load traces from an Australian smart grid trial (Solar Home Electricity Data), we demonstrate the efficacy of our method by comparing it with established energy- and peak demand-based cost allocation approaches.

eess.SY↗

arXiv2019

Energy management of small-scale PV-battery systems: A systematic review considering practical implementation, computational requirements, quality of input data and battery degradation

The home energy management problem has many different facets, including economic viability, data uncertainty and quality of strategy employed. The existing literature in this area focuses on individual aspects of this problem without a detailed, holistic analysis of the results with regards to practicality in implementation. In this paper, we fill this gap by performing a comprehensive comparison of seven different energy management strategies, each with different levels of practicality, sophistication and computational requirements. We analyse the results in the context of these three characteristics, and also critique the modelling assumptions made by each strategy. Our analysis finds that using a more sophisticated energy management strategy may not necessarily improve the performance and economic viability of the PV-battery system due to the effects of modeling assumptions, such as the treatment of uncertainties in the input data and battery degradation effects.

eess.SP↗

arXiv2018

A Nonparametric Bayesian Methodology for Synthesizing Residential Solar Generation and Demand Data

The uptake of behind-the-meter distributed energy resources in low-voltage distribution networks has reached a level where network issues have started to emerge, which requires new tools for operation and planning. In this paper, we propose a methodology for synthesizing stochastic demand and generation profiles for unobserved customers with rooftop PV, called prosumers. The proposed model bridges the gap between the limited available empirical data, and the large amount of high-quality, stochastic demand and generation data required for probabilistic analysis. The approach employs clustering analysis and a Dirichlet-categorical hierarchical model of the features of unobserved prosumers. Based on the data of clusters of prosumers, Markov chain models of demand and generation profiles are constructed from empirical data, and synthetic demand profiles are subsequently sampled from these. The sampled traces are cross-validated and show a good statistical fit to the observed data. Two case studies are considered to confirm the validity of the proposed methodology. The first studies the impact of behavioral differences on the synthetic demand profiles, while the second looks at the impact of varying solar generation penetration on demand profiles.

stat.AP↗

arXiv2018

Probabilistic assessment of the impact of flexible loads under network tariffs in low voltage distribution networks

Given the historically static nature of low-voltage networks, distribution network companies do not possess tools for dealing with an increasingly variable demand due to the high penetration of distributed energy resources (DER). Within this context, this paper proposes a probabilistic framework for tariff design that minimises the impact of DER on network performance, stabilise network company revenue, and improves the equity of network costs allocation. To address the issue of the lack of customers' response, we also show how DER-specific tariffs can be complemented with an automated home energy management system (HEMS) that reduces peak demand while retaining the desired comfort level. The proposed framework comprises a nonparametric Bayesian model which statistically generates synthetic load and PV traces, a hot-water-use statistical model, a novel HEMS to schedule customers' controllable devices, and a probabilistic power-flow model. Test cases using both energy- and demand-based network tariffs show that flat tariffs with a peak demand component reduce the customers' cost, and alleviate network constraints. This demonstrates, first, the efficacy of the proposed tool for the development of tariffs that are beneficial for networks with a high DER penetration, and second, how customers' HEM systems can be part of the solution.

eess.SY↗

arXiv2018

A Novel Probabilistic Framework to Study the Impact of PV-battery Systems on Low-Voltage Distribution Networks

Battery storage, particularly residential battery storage coupled with rooftop PV, is emerging as an essential component of the smart grid technology mix. However, including battery storage and other flexible resources like electric vehicles and loads with thermal inertia into a probabilistic analysis based on Monte Carlo (MC) simulation is challenging, because their operational profiles are determined by computationally intensive optimization. Additionally, MC analysis requires a large pool of statistically-representative demand profiles to sample from. As a result, the analysis of the network impact of PV-battery systems has attracted little attention in the existing literature. To fill these knowledge gaps, this paper proposes a novel probabilistic framework to study the impact of PV-battery systems on low-voltage distribution networks. Specifically, the framework incorporates home energy management(HEM) operational decisions within the MC time series power flow analysis. First, using available smart meter data, we use a Bayesian nonparametric model to generate statistically-representative synthetic demand and PV profiles. Second, a policy function approximation that emulates battery scheduling decisions is used to make the simulation of optimization-based HEM feasible within the MC framework. The efficacy of our method is demonstrated on three representative low-voltage feeders, where the computation time to execute our MC framework is 5% of that when using explicit optimization methods in each MC sample. The assessment results show that uncoordinated battery scheduling has a limited beneficial impact, which is against the conjecture that batteries will serendipitously mitigate the technical problems induced by PV generation.

eess.SY↗

arXiv2017

Generic Demand Model Considering the Impact of Prosumers for Future Grid Scenario Analysis

The increasing uptake of residential PV-battery systems is bound to significantly change demand patterns of future power systems and, consequently, their dynamic performance. In this paper, we propose a generic demand model that captures the aggregated effect of a large population of price-responsive users equipped with small-scale PV-battery systems, called prosumers, for market simulation in future grid scenario analysis. The model is formulated as a bi-level program in which the upper-level unit commitment problem minimizes the total generation cost, and the lower-level problem maximizes prosumers' aggregate self-consumption. Unlike in the existing bi-level optimization frameworks that focus on the interaction between the wholesale market and an aggregator, the coupling is through the prosumers' demand, not through the electricity price. That renders the proposed model market structure agnostic, making it suitable for future grid studies where the market structure is potentially unknown. As a case study, we perform steady-state voltage stability analysis of a simplified model of the Australian National Electricity Market with significant penetration of renewable generation. The simulation results show that a high prosumer penetration changes the demand profile in ways that significantly improve the system loadability, which confirms the suitability of the proposed model for future grid studies.

math.OC↗

arXiv2017

Computationally Efficient Market Simulation Tool for Future Grid Scenario Analysis

The paper proposes a computationally efficient electricity market simulation tool (MST) suitable for future grid scenario analysis. The market model is based on a unit commitment (UC) problem and takes into account the uptake of emerging technologies, like demand response, battery storage, concentrated solar thermal generation, and HVDC transmission lines. To allow for a subsequent stability assessment, the MST requires an explicit representation of the number of online generation units, which affects powers system inertia and reactive power support capability. These requirements render a fullfledged UC model computationally intractable, so we propose unit clustering, rolling horizon approach, and constraint reduction to increase the computational efficiency. To showcase the capability of the proposed tool, we use a simplified model of the Australian National Electricity Market with different penetrations of renewable generation. The results show that the number of online units resulting from the proposed tool is very close to the binary UC run over a week-long horizon, which is confirmed by the loadability and inertia analysis. That confirms the validity of the approach for long term future grid studies, where one is more interested in finding weak points in the system rather than in a detailed analysis of individual operating conditions.

math.OC↗

arXiv2016

A Distributed Algorithm for Demand Response with Mixed-Integer Variables

This letter presents a fast distributed algorithm for aggregating a large number of households with mixed-integer variables and intricate couplings between devices. The proposed fast distributed gradient algorithm is applied to the double smoothed dual function of the adopted DR model. The results also show that, with minimal parameter adjustments, the convergence of the dual objective exhibits the same behavior irrespective of the system size.

math.OC↗

arXiv2015

An Iterative On-Line Mechanism for Demand-Side Aggregation

This paper considers a demand-side aggregation scheme specifically for large numbers of small loads, such as households and small and medium-sized businesses. We introduce a novel auction format, called a staggered clock-proxy auction (SCPA), for on-line scheduling of these loads. This is a two phase format, consisting of: a sequence of overlapping iterative ascending-price clock auctions, one for each time-slot over a finite decision horizon, and; a set of proxy auctions that begin at the termination of each individual clock auction, and which determine the final price and allocation for each time-slot. The overlapping design of the clock phases grant bidders the ability to effectively bid on inter-temporal bundles of electricity use, thereby focusing on the most relevant parts of the price-quantity space. Since electricity is a divisible good, the proxy auction uses demand-schedule bids, which the aggregator uses to compute a uniform-price partial competitive equilibrium for each time slot. We show that, under mild assumptions on the bidders' utilities functions, the proxy phase implements the Vickrey-Clarke-Groves outcome, which makes straightforward bidding in the proxy phase a Bayes-Nash equilibrium. Furthermore, we demonstrate the SCPA in a scenario comprised of household agents with three different utility function types, and show how the mechanism enables efficient on-line energy use scheduling.

eess.SY↗

arXiv2012

Automated Planning in Repeated Adversarial Games

Game theory's prescriptive power typically relies on full rationality and/or self-play interactions. In contrast, this work sets aside these fundamental premises and focuses instead on heterogeneous autonomous interactions between two or more agents. Specifically, we introduce a new and concise representation for repeated adversarial (constant-sum) games that highlight the necessary features that enable an automated planing agent to reason about how to score above the game's Nash equilibrium, when facing heterogeneous adversaries. To this end, we present TeamUP, a model-based RL algorithm designed for learning and planning such an abstraction. In essence, it is somewhat similar to R-max with a cleverly engineered reward shaping that treats exploration as an adversarial optimization problem. In practice, it attempts to find an ally with which to tacitly collude (in more than two-player games) and then collaborates on a joint plan of actions that can consistently score a high utility in adversarial repeated games. We use the inaugural Lemonade Stand Game Tournament to demonstrate the effectiveness of our approach, and find that TeamUP is the best performing agent, demoting the Tournament's actual winning strategy into second place. In our experimental analysis, we show hat our strategy successfully and consistently builds collaborations with many different heterogeneous (and sometimes very sophisticated) adversaries.

cs.GT↗

arXiv2012

Filtered Fictitious Play for Perturbed Observation Potential Games and Decentralised POMDPs

Potential games and decentralised partially observable MDPs (Dec-POMDPs) are two commonly used models of multi-agent interaction, for static optimisation and sequential decisionmaking settings, respectively. In this paper we introduce filtered fictitious play for solving repeated potential games in which each player's observations of others' actions are perturbed by random noise, and use this algorithm to construct an online learning method for solving Dec-POMDPs. Specifically, we prove that noise in observations prevents standard fictitious play from converging to Nash equilibrium in potential games, which also makes fictitious play impractical for solving Dec-POMDPs. To combat this, we derive filtered fictitious play, and provide conditions under which it converges to a Nash equilibrium in potential games with noisy observations. We then use filtered fictitious play to construct a solver for Dec-POMDPs, and demonstrate our new algorithm's performance in a box pushing problem. Our results show that we consistently outperform the state-of-the-art Dec-POMDP solver by an average of 100% across the range of noise in the observation function.

cs.GT↗