arXiv subjects

Tom Brown

Publications and source records attributed to Tom Brown.

At least 55 records · Page 3Linked to original sources

arXiv2022

Constitutional AI: Harmlessness from AI Feedback

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.

cs.CL↗

arXiv2022

Meteorological conditions during Dunkelflauten in Germany: Characteristics, the role of weather regimes and impacts on demand

Renewable generation from wind and solar power is strongly weather-dependent. To plan future sustainable energy systems that are robust to this variability, a better understanding of why and when periods of low wind and solar power output occur is valuable. We call such periods of low wind and solar power output `Dunkelflauten', the German word for dark wind lulls. In this article, we analyse the meteorological conditions during Dunkelflauten in Germany by applying the concept of weather regimes. Weather regimes are quasi-stationary, recurrent, and persistent large-scale circulation patterns which explain multi-day atmospheric variability (5-15 days). We use a regime definition that allows us to distinguish four different types of blocked regimes, characterised by high pressure situations in the North Atlantic-European region. We find that in Germany, Dunkelflauten mainly occur in winter when the solar power output is anyway low and when the wind power output drops for several consecutive days. A high-pressure system over Germany, associated with the European Blocking regime, is responsible for most of the Dunkelflauten. Dunkelflauten during the Greenland Blocking regime are associated with colder temperatures than usual, causing higher electricity demand and presenting a particular challenge as space heating demand electrifies in future. Furthermore, we show that Dunkelflauten occur predominantly when a weather regime is well-established and persists longer than usual. Our study provides novel insight on the occurrence and meteorological characteristics of Dunkelflauten, which is essential for planning resilient energy systems and supporting grid operators to prepare for potential shortages in supply.

physics.ao-ph↗

arXiv2022

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.

cs.CL↗

arXiv2022

Language Models (Mostly) Know What They Know

We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.

cs.CL↗

arXiv2022

Measuring Progress on Scalable Oversight for Large Language Models

Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans and current general AI systems fail. We then present a proof-of-concept experiment meant to demonstrate a key feature of this experimental design and show its viability with two question-answering tasks: MMLU and time-limited QuALITY. On these tasks, we find that human participants who interact with an unreliable large-language-model dialog assistant through chat -- a trivial baseline strategy for scalable oversight -- substantially outperform both the model alone and their own unaided performance. These results are an encouraging sign that scalable oversight will be tractable to study with present models and bolster recent findings that large language models can productively assist humans with difficult tasks.

cs.HC↗

arXiv2022

Topology-based Approximations for $\mathcal{N}-1$ Contingency Constraints in Power Transmission Networks

It is crucial for maintaining the security of supply that transmission networks continue to operate even if a single line fails. Modeling $\mathcal{N} - 1$ security in power system capacity expansion problems introduces many extra constraints if all possible outages are accounted for, which leads to a high computational burden. Typical approaches to avoid this burden consider only a subset of possible outages relevant to a given dispatch situation. However, this relies on knowing the dispatch situation beforehand, and it is not suitable for investment optimization problems where the generation fleet is not known in advance. In this paper, we introduce a heuristic approach to model the fully secured $\mathcal{N}-1$ feasible space using a smaller number of constraints in a way that only depends on the topology of transmission networks. In our proposed approach, the network's security is modelled by comparing the polytope of the feasible space of nodal net power obtained from the security-constrained linearized AC optimal power flow problem. To approximate this polytope, a buffer capacity factor is defined for transmission lines in the $\mathcal{N}-0$ secure case, thereby avoiding the introduction of many additional constraints. In this way, three approaches are introduced for obtaining a buffer capacity factor consisting of approximate, robust and line-specific approaches. Finally, the performance of our proposed approaches is assessed in different scales of transmission networks for determining the proposed buffer capacity factors, contingency analysis and economic evaluation. Moreover, we find that our proposed heuristics provide excellent approximations of the fully secured $\mathcal{N}-1$ solutions with a much lower computational burden.

physics.soc-ph↗

arXiv2022

Predictability and Surprise in Large Generative Models

Large-scale pre-training has recently emerged as a technique for creating capable, general purpose, generative models such as GPT-3, Megatron-Turing NLG, Gopher, and many others. In this paper, we highlight a counterintuitive property of such models and discuss the policy implications of this property. Namely, these generative models have an unusual combination of predictable loss on a broad training distribution (as embodied in their "scaling laws"), and unpredictable specific capabilities, inputs, and outputs. We believe that the high-level predictability and appearance of useful capabilities drives rapid development of such models, while the unpredictable qualities make it difficult to anticipate the consequences of model deployment. We go through examples of how this combination can lead to socially harmful behavior with examples from the literature and real world observations, and we also perform two novel experiments to illustrate our point about harms from unpredictability. Furthermore, we analyze how these conflicting properties combine to give model developers various motivations for deploying these models, and challenges that can hinder deployment. We conclude with a list of possible interventions the AI community may take to increase the chance of these models having a beneficial impact. We intend this paper to be useful to policymakers who want to understand and regulate AI systems, technologists who care about the potential policy impact of their work, and academics who want to analyze, critique, and potentially develop large generative models.

cs.CY↗

arXiv2022

In-context Learning and Induction Heads

"Induction heads" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all "in-context learning" in large transformer models (i.e. decreasing loss at increasing token indices). We find that induction heads develop at precisely the same point as a sudden sharp increase in in-context learning ability, visible as a bump in the training loss. We present six complementary lines of evidence, arguing that induction heads may be the mechanistic source of general in-context learning in transformer models of any size. For small attention-only models, we present strong, causal evidence; for larger models with MLPs, we present correlational evidence.

cs.LG↗

arXiv2022

Are biofuel mandates cost-effective? -- an analysis of transport fuels and biomass usage to achieve emissions targets in the European energy system

Abatement options for the hard-to-electrify parts of the transport sector are needed to achieve ambitious emissions targets. Biofuels based on biomass, electrofuels based on renewable hydrogen and a carbon source, as well as fossil fuels compensated by carbon dioxide removal (CDR) are the main options. Currently, biofuels are the only renewable fuels available at scale and are stimulated by blending mandates. Here, we estimate the system cost of enforcing such mandates in addition to an overall emissions cap for all energy sectors. We model overnight scenarios for 2040 and 2060 with the sector-coupled European energy system model PyPSA-Eur-Sec, with a high temporal resolution. The following cost drivers are identified: (i) high biomass costs due to scarcity, (ii) opportunity costs for competing usages of biomass for industry heat and combined heat and power (CHP) with carbon capture, and (iii) lower scalability and generally higher cost for biofuels compared to electrofuels and fossil fuels combined with CDR. With a -80% emissions reduction target in 2040, variable renewables, partial electrification of heat, industry and transport and biomass use for CHP and industrial heat are important for achieving the target at minimal cost. Abatement of remaining liquid fossil fuel use increases system cost, with a 50% biofuel mandate increasing costs by 128-229 billion EUR, or 39-82% of the liquid fuel cost. With a negative -105% emissions target in 2060, fuel abatement options are necessary, and electrofuels or the use of CDR to offset fossil fuel emissions are more competitive than biofuels. Biomass is preferred in CHP and industry heat, combined with carbon capture to serve negative emissions or electrofuel production, thereby utilising biogenic carbon several times. Sensitivity analyses reveal significant uncertainties but consistently support that higher biofuel mandates lead to higher costs.

physics.soc-ph↗

arXiv2022

Scaling Laws and Interpretability of Learning from Repeated Data

Recent large language models have been trained on vast datasets, but also often on repeated data, either intentionally for the purpose of upweighting higher quality data, or unintentionally because data deduplication is not perfect and the model is exposed to repeated data at the sentence, paragraph, or document level. Some works have reported substantial negative performance effects of this repeated data. In this paper we attempt to study repeated data systematically and to understand its effects mechanistically. To do this, we train a family of models where most of the data is unique but a small fraction of it is repeated many times. We find a strong double descent phenomenon, in which repeated data can lead test loss to increase midway through training. A predictable range of repetition frequency leads to surprisingly severe degradation in performance. For instance, performance of an 800M parameter model can be degraded to that of a 2x smaller model (400M params) by repeating 0.1% of the data 100 times, despite the other 90% of the training tokens remaining unique. We suspect there is a range in the middle where the data can be memorized and doing so consumes a large fraction of the model's capacity, and this may be where the peak of degradation occurs. Finally, we connect these observations to recent mechanistic interpretability work - attempting to reverse engineer the detailed computations performed by the model - by showing that data repetition disproportionately damages copying and internal structures associated with generalization, such as induction heads, providing a possible mechanism for the shift from generalization to memorization. Taken together, these results provide a hypothesis for why repeating a relatively small fraction of data in large language models could lead to disproportionately large harms to performance.

cs.LG↗

arXiv2022

Long-Term Benefits of Network Boosters for Renewables Integration and Corrective Grid Security

The preventative strategies for $N-1$ network security dominant in European networks mean that network capacity is kept free in case a line fails. If instead fast corrective actions are used to overcome network overloading when single lines fail, this has the potential to free up network capacity that is otherwise underused in preventive $N-1$ security strategies. In this paper, we investigate the impact on renewable integration of a corrective network security strategy, whereby storage or other flexibility assets are used to correct overloading shortly after line outages. In this way, we find significant cost savings for the integration of renewable energy of up to 2.4 billion euros per year in an aggregated 50-bus model of the German power system utilizing these flexibility assets, so-called network boosters (NB). This offers a role for storage beyond energy arbitrage or ancillary services like frequency control. While previous literature has focused on the potential savings of NB in the short-term operation, we focus on the long-term benefits in systems with high shares of renewable energy sources, where the capacities and dispatch of generation and NB are optimised. We demonstrate the benefits of NB for various shares of renewable energy, NB and flexibility costs, as well as different allowed levels of temporary overloading the lines in both (i) a sequential model, where long-run generation investments are optimised separately from the NB capacities, and (ii) a simultaneous model, where generation is co-optimised with NB investment so that mixed preventive-corrective approaches are possible.

eess.SY↗

arXiv2022

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.

cs.CL↗

arXiv2022

Speed of technological transformations required in Europe to achieve different climate goals

Europe's contribution to global warming will be determined by the cumulative emissions until climate neutrality is achieved. In this paper, we investigate alternative transition paths under carbon budgets corresponding to temperature increases between 1.5 and 2C. We use PyPSA-Eur-Sec, an open model of the sector-coupled European energy system with high spatial and temporal resolution. All the paths entail similar technological transformations, but the timing of the scale-up of important technologies like water electrolysis, carbon capture and hydrogen networks differs in the model. In our results, solar PV, onshore and offshore wind become the cornerstone of a net-zero energy system enabling the decarbonisation of other sectors via direct electrification (e.g. heat pumps and electric vehicles) or indirect electrification (e.g. using synthetic fuels). Under the cost and performance assumptions applied, for a social cost of carbon (SCC) of 120EUR/tCO2, transition paths under 1.5 and 1.6C budgets are, respectively, 8%, and 1% more expensive than the 2C-budget because building assets earlier costs more. These pathways also see a faster ramp-up of new technologies before 2035. Under these assumptions, the 1.5C-budget is cost-optimal in our model, if SCC of at least 300 EUR/tCO2 is considered. Moreover, we discuss the strong implications of the SCC and discount rate assumed when comparing alternative paths. We also analyse the consequences of different assumptions on the cost and potential of CO2 sequestration.

physics.soc-ph↗

arXiv2021

A General Language Assistant as a Laboratory for Alignment

Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.

cs.CL↗

arXiv2021

Broad Ranges of Investment Configurations for Renewable Power Systems, Robust to Cost Uncertainty and Near-Optimality

To achieve ambitious greenhouse gas emission reduction targets in time, the planning of future energy systems needs to accommodate societal preferences, e.g. low levels of acceptance for transmission expansion or onshore wind turbines, and must also acknowledge the inherent uncertainties of technology cost projections. To date, however, many capacity expansion models lean heavily towards only minimising system cost and only studying a few cost projections. Here, we address both criticisms in unison. While taking account of technology cost uncertainties, we apply methods from multi-objective optimisation to explore trade-offs in a fully renewable European electricity system between increasing system cost and extremising the use of individual technologies for generating, storing and transmitting electricity to build robust insights about what actions are viable within given cost ranges. We identify boundary conditions that must be met for cost-efficiency regardless of how cost developments will unfold; for instance, that some grid reinforcement and long-term storage alongside a significant amount of wind capacity appear essential. But, foremost, we reveal that near the cost-optimum a broad spectrum of regionally and technologically diverse options exists in any case, which allows policymakers to navigate around public acceptance issues. The analysis requires managing many computationally demanding scenario runs efficiently, for which we leverage multi-fidelity surrogate modelling techniques using sparse polynomial chaos expansions and low-discrepancy sampling.

physics.soc-ph↗

arXiv2021

Extracting Training Data from Large Language Models

It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.

cs.CR↗

arXiv2021

Mitigating heat demand peaks in buildings in a highly renewable European energy system

Space and water heating accounts for about 40% of final energy consumption in the European Union and thus plays a key role in reducing overall costs and greenhouse gas emissions. Many scenarios to reach net-zero emissions in buildings rely on electrification, but meeting the heat demand peaks in the winter can be challenging, particularly when wind and solar resources are low. This paper examines how to mitigate space heating demand peaks most cost-effectively in a top-down, sector-coupled model with carbon dioxide emissions constraint to be net-zero. It introduces the first model that co-optimises both supply and efficiency simultaneously including all European countries with hourly resolution. The competition between technologies to address these heating peaks, namely building retrofitting, thermal energy storage and individual hybrid heat pumps with backup gas boilers is examined. A novel thought experiment demonstrates that the level of building renovation is driven by the strong seasonal heat peaks, rather than the overall energy consumption. If all three instruments are applied, total costs are reduced by up to 17%. Building renovation enables the largest benefit with cost savings of up to 14% and allows individual gas boilers to be removed from the energy system without significant higher costs.

physics.soc-ph↗

arXiv2020

Early decarbonisation of the European energy system pays off

For a given carbon budget over several decades, different transformation rates for the energy system yield starkly different results. Here we consider a budget of 33 GtCO2 for the cumulative carbon dioxide emissions from the European electricity, heating, and transport sectors between 2020 and 2050, which represents Europe's contribution to the Paris Agreement. We have found that following an early and steady path in which emissions are strongly reduced in the first decade is more cost-effective than following a late and rapid path in which low initial reduction targets quickly deplete the carbon budget and require a sharp reduction later. We show that solar photovoltaic, onshore and offshore wind can become the cornerstone of a fully decarbonised energy system and that installation rates similar to historical maxima are required to achieve timely decarbonisation. Key to those results is a proper representation of existing balancing strategies through an open, hourly-resolved, networked model of the sector-coupled European energy system.

physics.soc-ph↗