arXiv subjects

Andrew Gallagher

Publications and source records attributed to Andrew Gallagher.

10 recordsLinked to original sources

arXiv2024

Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%.

cs.CV↗

arXiv2023

Building One-class Detector for Anything: Open-vocabulary Zero-shot OOD Detection Using Text-image Models

We focus on the challenge of out-of-distribution (OOD) detection in deep learning models, a crucial aspect in ensuring reliability. Despite considerable effort, the problem remains significantly challenging in deep learning models due to their propensity to output over-confident predictions for OOD inputs. We propose a novel one-class open-set OOD detector that leverages text-image pre-trained models in a zero-shot fashion and incorporates various descriptions of in-domain and OOD. Our approach is designed to detect anything not in-domain and offers the flexibility to detect a wide variety of OOD, defined via fine- or coarse-grained labels, or even in natural language. We evaluate our approach on challenging benchmarks including large-scale datasets containing fine-grained, semantically similar classes, distributionally shifted images, and multi-object images containing a mixture of in-domain and OOD objects. Our method shows superior performance over previous methods on all benchmarks. Code is available at https://github.com/gyhandy/One-Class-Anything

cs.CV↗

arXiv2022

Improving Zero-shot Generalization and Robustness of Multi-modal Models

Multi-modal image-text models such as CLIP and LiT have demonstrated impressive performance on image classification benchmarks and their zero-shot generalization ability is particularly exciting. While the top-5 zero-shot accuracies of these models are very high, the top-1 accuracies are much lower (over 25% gap in some cases). We investigate the reasons for this performance gap and find that many of the failure cases are caused by ambiguity in the text prompts. First, we develop a simple and efficient zero-shot post-hoc method to identify images whose top-1 prediction is likely to be incorrect, by measuring consistency of the predictions w.r.t. multiple prompts and image transformations. We show that our procedure better predicts mistakes, outperforming the popular max logit baseline on selective prediction tasks. Next, we propose a simple and efficient way to improve accuracy on such uncertain images by making use of the WordNet hierarchy; specifically we augment the original class by incorporating its parent and children from the semantic label hierarchy, and plug the augmentation into text prompts. We conduct experiments on both CLIP and LiT models with five different ImageNet-based datasets. For CLIP, our method improves the top-1 accuracy by 17.13% on the uncertain subset and 3.6% on the entire ImageNet validation set. We also show that our method improves across ImageNet shifted datasets, four other datasets, and other model architectures such as LiT. The proposed method is hyperparameter-free, requires no additional model training and can be easily scaled to other large multi-modal architectures. Code is available at https://github.com/gyhandy/Hierarchy-CLIP.

cs.CV↗

arXiv2020

The Gaia-ESO survey: 3D NLTE abundances in the open cluster NGC 2420 suggest atomic diffusion and turbulent mixing at the origin of chemical abundance variations

Atomic diffusion and mixing processes in stellar interiors influence the structure and the surface composition of stars. Some of these processes cannot yet be modelled from the first principles. This limits their applicability in stellar models used for studies of stellar populations and Galactic evolution. Our main goal is to put constrains on the stellar structure and evolution models using new refined measurements of chemical composition in stars of Galactic open cluster. We use medium-resolution, 19 200 <= R <= 21 500, optical spectra of the stars in the open cluster NGC 2420 obtained within the Gaia-ESO survey. The sample covers all evolutionary stages from the main-sequence to red giant branch. Stellar parameters are derived using a combined Bayesian analysis of spectra, 2MASS photometry, and astrometric data from Gaia DR2. The abundances of Mg, Ca, Fe, and Li are determined from non-local thermodynamic equilibrium (NLTE) synthetic spectra, computed using one-dimensional (1D) and averaged three-dimensional (3D) model atmospheres. We compare our results with a grid of Code d'Evolution Stellaire Adaptatif et Modulaire (CESTAM) stellar evolution models, which include atomic diffusion, turbulent and rotational mixing. We find prominent evolutionary trends in the abundances of Fe, Ca, Mg, and Li with the mass of the stars in the cluster. Fe, Mg, and Ca show a depletion at the cluster turn-off, but the abundances gradually increase and flatten near the base of the RGB. The abundance trend for Li displays a signature of rotational mixing on the main-sequence and abrupt depletion on the subgiant branch, which is caused by advection of Li-poor material to the surface. The analysis of abundances combined with the CESTAM model predictions allows us to place limits on the parameter space of the models and to constrain the zone in the stellar interior where turbulent mixing takes place.

astro-ph.SR↗

arXiv2020

Automatic Differentiation Variational Inference with Mixtures

Automatic Differentiation Variational Inference (ADVI) is a useful tool for efficiently learning probabilistic models in machine learning. Generally approximate posteriors learned by ADVI are forced to be unimodal in order to facilitate use of the reparameterization trick. In this paper, we show how stratified sampling may be used to enable mixture distributions as the approximate posterior, and derive a new lower bound on the evidence analogous to the importance weighted autoencoder (IWAE). We show that this "SIWAE" is a tighter bound than both IWAE and the traditional ELBO, both of which are special instances of this bound. We verify empirically that the traditional ELBO objective disfavors the presence of multimodal posterior distributions and may therefore not be able to fully capture structure in the latent space. Our experiments show that using the SIWAE objective allows the encoder to learn more complex distributions which regularly contain multimodality, resulting in higher accuracy and better calibration in the presence of incomplete, limited, or corrupted data.

cs.LG↗

arXiv2019

AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection

Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual dataset for this task has constrained algorithm evaluations with respect to data diversity, environments, and accuracy. This has made comparisons and improvements difficult. In this paper, we present the AVA Active Speaker detection dataset (AVA-ActiveSpeaker) that will be released publicly to facilitate algorithm development and enable comparisons. The dataset contains temporally labeled face tracks in video, where each face instance is labeled as speaking or not, and whether the speech is audible. This dataset contains about 3.65 million human labeled frames or about 38.5 hours of face tracks, and the corresponding audio. We also present a new audio-visual approach for active speaker detection, and analyze its performance, demonstrating both its strength and the contributions of the dataset.

cs.CV↗

arXiv2018

Modeling Uncertainty with Hedged Instance Embedding

Instance embeddings are an efficient and versatile image representation that facilitates applications like recognition, verification, retrieval, and clustering. Many metric learning methods represent the input as a single point in the embedding space. Often the distance between points is used as a proxy for match confidence. However, this can fail to represent uncertainty arising when the input is ambiguous, e.g., due to occlusion or blurriness. This work addresses this issue and explicitly models the uncertainty by hedging the location of each input in the embedding space. We introduce the hedged instance embedding (HIB) in which embeddings are modeled as random variables and the model is trained under the variational information bottleneck principle. Empirical results on our new N-digit MNIST dataset show that our method leads to the desired behavior of hedging its bets across the embedding space upon encountering ambiguous inputs. This results in improved performance for image matching and classification tasks, more structure in the learned embedding space, and an ability to compute a per-exemplar uncertainty measure that is correlated with downstream performance.

cs.LG↗

arXiv2018

AVA-Speech: A Densely Labeled Dataset of Speech Activity in Movies

Speech activity detection (or endpointing) is an important processing step for applications such as speech recognition, language identification and speaker diarization. Both audio- and vision-based approaches have been used for this task in various settings, often tailored toward end applications. However, much of the prior work reports results in synthetic settings, on task-specific datasets, or on datasets that are not openly available. This makes it difficult to compare approaches and understand their strengths and weaknesses. In this paper, we describe a new dataset which we will release publicly containing densely labeled speech activity in YouTube videos, with the goal of creating a shared, available dataset for this task. The labels in the dataset annotate three different speech activity conditions: clean speech, speech co-occurring with music, and speech co-occurring with noise, which enable analysis of model performance in more challenging conditions based on the presence of overlapping noise. We report benchmark performance numbers on AVA-Speech using off-the-shelf, state-of-the-art audio and vision models that serve as a baseline to facilitate future research.

cs.SD↗

arXiv2018

Finding your Lookalike: Measuring Face Similarity Rather than Face Identity

Face images are one of the main areas of focus for computer vision, receiving on a wide variety of tasks. Although face recognition is probably the most widely researched, many other tasks such as kinship detection, facial expression classification and facial aging have been examined. In this work we propose the new, subjective task of quantifying perceived face similarity between a pair of faces. That is, we predict the perceived similarity between facial images, given that they are not of the same person. Although this task is clearly correlated with face recognition, it is different and therefore justifies a separate investigation. Humans often remark that two persons look alike, even in cases where the persons are not actually confused with one another. In addition, because face similarity is different than traditional image similarity, there are challenges in data collection and labeling, and dealing with diverging subjective opinions between human labelers. We present evidence that finding facial look-alikes and recognizing faces are two distinct tasks. We propose a new dataset for facial similarity and introduce the Lookalike network, directed towards similar face classification, which outperforms the ad hoc usage of a face recognition network directed at the same task.

cs.CV↗

arXiv2002

Arm Structure in Anemic Spiral Galaxies

Anemic galaxies have less prominent star formation than normal galaxies of the same Hubble type. Previous studies showed they are deficient in total atomic hydrogen but not in molecular hydrogen. Here we compare the combined surface densities of HI and H2 at mid-disk radii with the Kennicutt threshold for star formation. The anemic galaxies are below threshold, which explains their lack of prominent star formation, but they are not much different than other early type galaxies, which also tend to be below threshold. The spiral wave amplitudes of anemic and normal galaxies were also compared, using images in B and J passbands from the OSU Bright Spiral Galaxy Survey. Anemic galaxies have normal spiral wave properties too, with the same amplitudes and radial dependencies as other galaxies of the same arm class. Because of the lack of gas, spiral waves in early type galaxies and anemics do not have a continuous supply of stars with low velocity dispersions to maintain a marginally stable disk. As a result, they are either short-lived, evolving toward lenticulars and S0 types in only a few rotations at mid-disk, or they are driven by the asymmetries associated with gas removal in the cluster environment.

astro-ph↗