arXiv subjects

Ellie Kitanidis

Publications and source records attributed to Ellie Kitanidis.

8 recordsLinked to original sources

arXiv2023

Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training

Hyperparameter tuning of deep learning models can lead to order-of-magnitude performance gains for the same amount of compute. Despite this, systematic tuning is uncommon, particularly for large models, which are expensive to evaluate and tend to have many hyperparameters, necessitating difficult judgment calls about tradeoffs, budgets, and search bounds. To address these issues and propose a practical method for robustly tuning large models, we present Cost-Aware Pareto Region Bayesian Search (CARBS), a Bayesian optimization algorithm that performs local search around the performance-cost Pareto frontier. CARBS does well even in unbounded search spaces with many hyperparameters, learns scaling relationships so that it can tune models even as they are scaled up, and automates much of the "black magic" of tuning. Among our results, we effectively solve the entire ProcGen benchmark just by tuning a simple baseline (PPO, as provided in the original ProcGen paper). We also reproduce the model size vs. training tokens scaling result from the Chinchilla project (Hoffmann et al. 2022), while simultaneously discovering scaling laws for every other hyperparameter, via an easy automated process that uses significantly less compute and is applicable to any deep learning problem (not just language models).

cs.LG↗

arXiv2022

Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safety

Large language models (LLMs) have exploded in popularity in the past few years and have achieved undeniably impressive results on benchmarks as varied as question answering and text summarization. We provide a simple new prompting strategy that leads to yet another supposedly "super-human" result, this time outperforming humans at common sense ethical reasoning (as measured by accuracy on a subset of the ETHICS dataset). Unfortunately, we find that relying on average performance to judge capabilities can be highly misleading. LLM errors differ systematically from human errors in ways that make it easy to craft adversarial examples, or even perturb existing examples to flip the output label. We also observe signs of inverse scaling with model size on some examples, and show that prompting models to "explain their reasoning" often leads to alarming justifications of unethical actions. Our results highlight how human-like performance does not necessarily imply human-like understanding or reasoning.

cs.CL↗

arXiv2022

Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds

Despite impressive successes, deep reinforcement learning (RL) systems still fall short of human performance on generalization to new tasks and environments that differ from their training. As a benchmark tailored for studying RL generalization, we introduce Avalon, a set of tasks in which embodied agents in highly diverse procedural 3D worlds must survive by navigating terrain, hunting or gathering food, and avoiding hazards. Avalon is unique among existing RL benchmarks in that the reward function, world dynamics, and action space are the same for every task, with tasks differentiated solely by altering the environment; its 20 tasks, ranging in complexity from eat and throw to hunt and navigate, each create worlds in which the agent must perform specific skills in order to survive. This setup enables investigations of generalization within tasks, between tasks, and to compositional tasks that require combining skills learned from previous tasks. Avalon includes a highly efficient simulator, a library of baselines, and a benchmark with scoring metrics evaluated against hundreds of hours of human performance, all of which are open-source and publicly available. We find that standard RL baselines make progress on most tasks but are still far from human performance, suggesting Avalon is challenging enough to advance the quest for generalizable RL.

cs.AI↗

arXiv2021

Preliminary clustering properties of the DESI BGS bright targets using DR9 Legacy Imaging Surveys

We characterise the selection cuts and clustering properties of a magnitude-limited sample of bright galaxies that is part of the Bright Galaxy Survey (BGS) of the Dark Energy Spectroscopic Instrument (DESI) using the ninth data release of the Legacy Imaging Surveys (DR9). We describe changes in the DR9 selection compared to the DR8 one as explored in Ruiz-Macias et al. (2021). We also compare the DR9 selection in three distinct regions: BASS/MzLS in the north Galactic Cap (NGC), DECaLS in the NGC, and DECaLS in the south Galactic Cap (SGC). We investigate the systematics associated with the selection and assess its completeness by matching the BGS targets with the Galaxy and Mass Assembly (GAMA) survey. We measure the angular clustering for the overall bright sample (r $\leq$ 19.5) and as function of apparent magnitude and colour. This enables to determine the clustering strength and slope by fitting a power-law model that can be used to generate accurate mock catalogues for this tracer. We use a counts-in-cells technique to explore higher-order statistics and cross-correlations with external spectroscopic data sets in order to check the evolution of the clustering with redshift and the redshift distribution of the BGS targets using clustering-redshifts. While this work validates the properties of the BGS bright targets, the final target selection pipeline and clustering properties of the entire DESI BGS will be fully characterised and validated with the spectroscopic data of Survey Validation.

astro-ph.GA↗

arXiv2021

Characterising the target selection pipeline for the Dark Energy Spectroscopic Instrument Bright Galaxy Survey

We present the steps taken to produce a reliable and complete input galaxy catalogue for the Dark Energy Spectroscopic Instrument (DESI) Bright Galaxy Survey (BGS) using the photometric Legacy Survey DR8 DECam. We analyse some of the main issues faced in the selection of targets for the DESI BGS, such as star-galaxy separation, contamination by fragmented stars and bright galaxies. Our pipeline utilizes a new way to select BGS galaxies using Gaia photometry and we implement geometrical and photometric masks that reduce the number of spurious objects. The resulting catalogue is cross-matched with the Galaxy And Mass Assembly (GAMA) survey to assess the completeness of the galaxy catalogue and the performance of the target selection. We also validate the clustering of the sources in our BGS catalogue by comparing with mock catalogues and the Sloan Digital Sky Survey (SDSS) data. Finally, the robustness of the BGS selection criteria is assessed by quantifying the dependence of the target galaxy density on imaging and other properties. The largest systematic correlation we find is a 7 per cent suppression of the target density in regions of high stellar density.

astro-ph.GA↗

arXiv2020

Cross-Correlation of Planck CMB Lensing with DESI-Like LRGs

Cross-correlations between the lensing of the cosmic microwave background (CMB) and other tracers of large-scale structure provide a unique way to reconstruct the growth of dark matter, break degeneracies between cosmology and galaxy physics, and test theories of modified gravity. We detect a cross-correlation between DESI-like luminous red galaxies (LRGs) selected from DECaLS imaging and CMB lensing maps reconstructed with the Planck satellite at a significance of $S/N = 27.2$ over scales $\ell_{\rm min} = 30$, $\ell_{\rm max} = 1000$. To correct for magnification bias, we determine the slope of the LRG cumulative magnitude function at the faint limit as $s = 0.999 \pm 0.015$, and find corresponding corrections on the order of a few percent for $C^{κg}_{\ell}, C^{gg}_{\ell}$ across the scales of interest. We fit the large-scale galaxy bias at the effective redshift of the cross-correlation $z_{\rm eff} \approx 0.68$ using two different bias evolution agnostic models: a HaloFit times linear bias model where the bias evolution is folded into the clustering-based estimation of the redshift kernel, and a Lagrangian perturbation theory model of the clustering evaluated at $z_{\rm eff}$. We also determine the error on the bias from uncertainty in the redshift distribution; within this error, the two methods show excellent agreement with each other and with DESI survey expectations.

astro-ph.CO↗

arXiv2020

Imaging Systematics and Clustering of DESI Main Targets

We evaluate the impact of imaging systematics on the clustering of luminous red galaxies (LRG), emission-line galaxies (ELG) and quasars (QSO) targeted for the upcoming Dark Energy Spectroscopic Instrument (DESI) survey. Using Data Release 7 of the DECam Legacy Survey, we study the effects of astrophysical foregrounds, stellar contamination, differences between north galactic cap and south galactic cap measurements, and variations in imaging depth, stellar density, galactic extinction, seeing, airmass, sky brightness, and exposure time before presenting survey masks and weights to mitigate these effects. With our sanitized samples in hand, we conduct a preliminary analysis of the clustering amplitude and evolution of the DESI main targets. From measurements of the angular correlation functions, we determine power law fits $r_0 = 7.78 \pm 0.26$ $h^{-1}$Mpc, $γ= 1.98 \pm 0.02$ for LRGs and $r_0 = 5.45 \pm 0.1$ $h^{-1}$Mpc, $γ= 1.54 \pm 0.01$ for ELGs. Additionally, from the angular power spectra, we measure the linear biases and model the scale dependent biases in the weakly nonlinear regime. Both sets of clustering measurements show good agreement with survey requirements for LRGs and ELGs, attesting that these samples will enable DESI to achieve precise cosmological constraints. We also present clustering as a function of magnitude, use cross-correlations with external spectroscopy to infer $dN/dz$ and measure clustering as a function of luminosity, and probe higher order clustering statistics through counts-in-cells moments.

astro-ph.CO↗

arXiv2015

Cosmological Evolution of Long Gamma-ray Bursts and Star Formation Rate

Gamma-ray bursts (GRBs) by virtue of their high luminosities can be detected up to very high redshifts and therefore can be excellent probes of the early universe. This task is hampered by the fact that most of their characteristics have a broad range so that we first need to obtain an accurate description of the distribution of these characteristics, and specially, their cosmological evolution. We use a sample of about 200 \swift long GRBs with known redshift to determine the luminosity and formation rate evolutions and the general shape of the luminosity function. In contrast to most other forward fitting methods of treating this problem we use the Efron Petrosian methods which allow a non-parametric determination of above quantities. We find a relatively strong luminosity evolution, a luminosity function that can be fitted to a broken power law, and an unusually high rate of formation rate at low redshifts, a rate more than one order of magnitude higher than the star formation rate (SFR). On the other hand, our results seem to agree with the almost constant SFR in redshifts 1 to 3 and the decline above this redshift.

astro-ph.HE↗