arXiv subjects

Patrick Rehill

Publications and source records attributed to Patrick Rehill.

8 recordsLinked to original sources

arXiv2026

A rapid evaluation of Australia's COVID-era apprentice wage subsidy programs

In the midst of the COVID-19 pandemic in 2020, the Australian Government launched two programs to incentivise new apprentices to start and complete apprenticeships -- the Boosting Apprenticeship Commencements (BAC) and Completing Apprenticeship Commencements (CAC) programs. These programs were wage subsidies to encourage employers to take on or retain apprentices. This paper evaluates the impact of these programs on apprenticeship commencements and completions taking a mixed-methods approach combining econometric modelling and interviews with stakeholders including employers and peak bodies. The programs led to a 70\% increase in commencement of apprenticeships but do not seem to have boosted retention rates. There appears to be a small increase in cancellation rates suggesting lower eventual completion rates compared to previous cohorts. Cancellation rates were higher for non-trade commencements (7\% increase) during BAC, but slightly lower for trade commencements (0.7\% decrease). We find this effect in non-trade apprenticeships was likely driven by `sharp practice' where some employers took advantage of the BAC by converting existing employees over to apprenticeships to attract the wage subsidy with no intention of having these employees stay as apprentices beyond the period of the BAC's generous subsidy. While the BAC / CAC were successful in many of their goals, there are several lessons that can be learnt from its design. In particular, the need to implement the program quickly meant early design choices inadvertently encouraged `sharp practice' and a rush for places that placed strain on the training sector. However, employers appreciated the front-loading of payments which provided the most financial support when apprentices were new and at their least productive.

econ.GN↗

arXiv2024

Distilling interpretable causal trees from causal forests

Machine learning methods for estimating treatment effect heterogeneity promise greater flexibility than existing methods that test a few pre-specified hypotheses. However, one problem these methods can have is that it can be challenging to extract insights from complicated machine learning models. A high-dimensional distribution of conditional average treatment effects may give accurate, individual-level estimates, but it can be hard to understand the underlying patterns; hard to know what the implications of the analysis are. This paper proposes the Distilled Causal Tree, a method for distilling a single, interpretable causal tree from a causal forest. This compares well to existing methods of extracting a single tree, particularly in noisy data or high-dimensional data where there are many correlated features. Here it even outperforms the base causal forest in most simulations. Its estimates are doubly robust and asymptotically normal just as those of the causal forest are.

econ.EM↗

arXiv2024

How do applied researchers use the Causal Forest? A methodological review of a method

This methodological review examines the use of the causal forest method by applied researchers across 133 peer-reviewed papers. It shows that the emerging best practice relies heavily on the approach and tools created by the original authors of the causal forest such as their grf package and the approaches given by them in examples. Generally researchers use the causal forest on a relatively low-dimensional dataset relying on observed controls or in some cases experiments to identify effects. There are several common ways to then communicate results -- by mapping out the univariate distribution of individual-level treatment effect estimates, displaying variable importance results for the forest and graphing the distribution of treatment effects across covariates that are important either for theoretical reasons or because they have high variable importance. Some deviations from this common practice are interesting and deserve further development and use. Others are unnecessary or even harmful. The paper concludes by reflecting on the emerging best practice for causal forest use and paths for future research.

econ.EM↗

arXiv2024

Heterogeneous treatment effect estimation with high-dimensional data in public policy evaluation -- an application to the conditioning of cash transfers in Morocco using causal machine learning

Causal machine learning methods can be used to search for treatment effect heterogeneity in high-dimensional datasets even where we lack a strong enough theoretical framework to select variables or make parametric assumptions about data. This paper uses causal machine learning methods to estimate heterogeneous treatment effects in the case of an experimental study carried out in Morocco which evaluated the effect of conditionalizing a cash transfer program on several outcomes including maths test scores which is the focus of this work. We explore treatment effects across a dataset of 1936 pre-treatment variables. For the most part, heterogeneity is modelled by two different factors, participation in education (at the baseline) and more general measures of poverty. Those who are more disadvantaged at the baseline benefit less from any treatment. While conditioning generally has a negative effect this more disadvantaged group is also hurt more by conditioning. The second purpose of this paper is to demonstrate and reflect upon a causal machine learning approach to policy evaluation. We propose a novel causal tree method for interpretable modelling of causal effects and reflect on the difficulty of explaining atheoretical results.

econ.GN↗

arXiv2023

Transparency challenges in policy evaluation with causal machine learning -- improving usability and accountability

Causal machine learning tools are beginning to see use in real-world policy evaluation tasks to flexibly estimate treatment effects. One issue with these methods is that the machine learning models used are generally black boxes, i.e., there is no globally interpretable way to understand how a model makes estimates. This is a clear problem in policy evaluation applications, particularly in government, because it is difficult to understand whether such models are functioning in ways that are fair, based on the correct interpretation of evidence and transparent enough to allow for accountability if things go wrong. However, there has been little discussion of transparency problems in the causal machine learning literature and how these might be overcome. This paper explores why transparency issues are a problem for causal machine learning in public policy evaluation applications and considers ways these problems might be addressed through explainable AI tools and by simplifying models in line with interpretable AI principles. It then applies these ideas to a case-study using a causal forest model to estimate conditional average treatment effects for a hypothetical change in the school leaving age in Australia. It shows that existing tools for understanding black-box predictive models are poorly suited to causal machine learning and that simplifying the model to make it interpretable leads to an unacceptable increase in error (in this application). It concludes that new tools are needed to properly understand causal machine learning models and the algorithms that fit them.

cs.LG↗

arXiv2023

Fairness Implications of Heterogeneous Treatment Effect Estimation with Machine Learning Methods in Policy-making

Causal machine learning methods which flexibly generate heterogeneous treatment effect estimates could be very useful tools for governments trying to make and implement policy. However, as the critical artificial intelligence literature has shown, governments must be very careful of unintended consequences when using machine learning models. One way to try and protect against unintended bad outcomes is with AI Fairness methods which seek to create machine learning models where sensitive variables like race or gender do not influence outcomes. In this paper we argue that standard AI Fairness approaches developed for predictive machine learning are not suitable for all causal machine learning applications because causal machine learning generally (at least so far) uses modelling to inform a human who is the ultimate decision-maker while AI Fairness approaches assume a model that is making decisions directly. We define these scenarios as indirect and direct decision-making respectively and suggest that policy-making is best seen as a joint decision where the causal machine learning model usually only has indirect power. We lay out a definition of fairness for this scenario - a model that provides the information a decision-maker needs to accurately make a value judgement about just policy outcomes - and argue that the complexity of causal machine learning models can make this difficult to achieve. The solution here is not traditional AI Fairness adjustments, but careful modelling and awareness of some of the decision-making biases that these methods might encourage which we describe.

econ.EM↗

arXiv2023

Counterfactually Fair Regression with Double Machine Learning

Counterfactual fairness is an approach to AI fairness that tries to make decisions based on the outcomes that an individual with some kind of sensitive status would have had without this status. This paper proposes Double Machine Learning (DML) Fairness which analogises this problem of counterfactual fairness in regression problems to that of estimating counterfactual outcomes in causal inference under the Potential Outcomes framework. It uses arbitrary machine learning methods to partial out the effect of sensitive variables on nonsensitive variables and outcomes. Assuming that the effects of the two sets of variables are additively separable, outcomes will be approximately equalised and individual-level outcomes will be counterfactually fair. This paper demonstrates the approach in a simulation study pertaining to discrimination in workplace hiring and an application on real data estimating the GPAs of law school students. It then discusses when it is appropriate to apply such a method to problems of real-world discrimination where constructs are conceptually complex and finally, whether DML Fairness can achieve justice in these settings.

cs.LG↗

arXiv2022

Policy learning for many outcomes of interest: Combining optimal policy trees with multi-objective Bayesian optimisation

Methods for learning optimal policies use causal machine learning models to create human-interpretable rules for making choices around the allocation of different policy interventions. However, in realistic policy-making contexts, decision-makers often care about trade-offs between outcomes, not just single-mindedly maximising utility for one outcome. This paper proposes an approach termed Multi-Objective Policy Learning (MOPoL) which combines optimal decision trees for policy learning with a multi-objective Bayesian optimisation approach to explore the trade-off between multiple outcomes. It does this by building a Pareto frontier of non-dominated models for different hyperparameter settings which govern outcome weighting. The key here is that a low-cost greedy tree can be an accurate proxy for the very computationally costly optimal tree for the purposes of making decisions which means models can be repeatedly fit to learn a Pareto frontier. The method is applied to a real-world case-study of non-price rationing of anti-malarial medication in Kenya.

cs.LG↗