arXiv subjects

Joshua Robinson

Publications and source records attributed to Joshua Robinson.

At least 37 records · Page 2Linked to original sources

arXiv2023

Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning

Self-supervised learning converts raw perceptual data such as images to a compact space where simple Euclidean distances measure meaningful variations in data. In this paper, we extend this formulation by adding additional geometric structure to the embedding space by enforcing transformations of input space to correspond to simple (i.e., linear) transformations of embedding space. Specifically, in the contrastive learning setting, we introduce an equivariance objective and theoretically prove that its minima forces augmentations on input space to correspond to rotations on the spherical embedding space. We show that merely combining our equivariant loss with a non-collapse term results in non-trivial representations, without requiring invariance to data augmentations. Optimal performance is achieved by also encouraging approximate invariance, where input augmentations correspond to small rotations. Our method, CARE: Contrastive Augmentation-induced Rotational Equivariance, leads to improved performance on downstream tasks, and ensures sensitivity in embedding space to important variations in data (e.g., color) that standard contrastive methods do not achieve. Code is available at https://github.com/Sharut/CARE.

cs.LG↗

arXiv2023

Leveraging Large Language Models for Multiple Choice Question Answering

While large language models (LLMs) like GPT-3 have achieved impressive results on multiple choice question answering (MCQA) tasks in the zero, one, and few-shot settings, they generally lag behind the MCQA state of the art (SOTA). MCQA tasks have traditionally been presented to LLMs like cloze tasks. An LLM is conditioned on a question (without the associated answer options) and its chosen option is the one assigned the highest probability after normalization (for length, etc.). A more natural prompting approach is to present the question and answer options to the LLM jointly and have it output the symbol (e.g., "A") associated with its chosen answer option. This approach allows the model to explicitly compare answer options, reduces computational costs, and mitigates the effects of tokenization scheme and answer option representations on answer selection. For the natural approach to be effective, the LLM it is used with must be able to associate answer options with the symbols that represent them. The LLM needs what we term multiple choice symbol binding (MCSB) ability. This ability varies greatly by model. We show that a model with high MCSB ability performs much better with the natural approach than with the traditional approach across 20 diverse datasets and largely closes the gap with the SOTA, suggesting that the MCQA ability of LLMs has been previously underestimated.

cs.CL↗

arXiv2023

A deep learning approach to using wearable seismocardiography (SCG) for diagnosing aortic valve stenosis and predicting aortic hemodynamics obtained by 4D flow MRI

In this paper, we explored the use of deep learning for the prediction of aortic flow metrics obtained using 4D flow MRI using wearable seismocardiography (SCG) devices. 4D flow MRI provides a comprehensive assessment of cardiovascular hemodynamics, but it is costly and time-consuming. We hypothesized that deep learning could be used to identify pathological changes in blood flow, such as elevated peak systolic velocity Vmax in patients with heart valve diseases, from SCG signals. We also investigated the ability of this deep learning technique to differentiate between patients diagnosed with aortic valve stenosis (AS), non-AS patients with a bicuspid aortic valve (BAV), non-AS patients with a mechanical aortic valve (MAV), and healthy subjects with a normal tricuspid aortic valve (TAV). In a study of 77 subjects who underwent same-day 4D flow MRI and SCG, we found that the Vmax values obtained using deep learning and SCGs were in good agreement with those obtained by 4D flow MRI. Additionally, subjects with TAV, BAV, MAV, and AS could be classified with ROC-AUC values of 92%, 95%, 81%, and 83%, respectively. This suggests that SCG obtained using low-cost wearable electronics may be used as a supplement to 4D flow MRI exams or as a screening tool for aortic valve disease.

cs.LG↗

arXiv2022

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions

Integrating functions on discrete domains into neural networks is key to developing their capability to reason about discrete objects. But, discrete domains are (1) not naturally amenable to gradient-based optimization, and (2) incompatible with deep learning architectures that rely on representations in high-dimensional vector spaces. In this work, we address both difficulties for set functions, which capture many important discrete problems. First, we develop a framework for extending set functions onto low-dimensional continuous domains, where many extensions are naturally defined. Our framework subsumes many well-known extensions as special cases. Second, to avoid undesirable low-dimensional neural network bottlenecks, we convert low-dimensional extensions into representations in high-dimensional spaces, taking inspiration from the success of semidefinite programs for combinatorial optimization. Empirically, we observe benefits of our extensions for unsupervised neural combinatorial optimization, in particular with high-dimensional representations.

cs.LG↗

arXiv2022

A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

We introduce CAN, a simple, efficient and scalable method for self-supervised learning of visual representations. Our framework is a minimal and conceptually clean synthesis of (C) contrastive learning, (A) masked autoencoders, and (N) the noise prediction approach used in diffusion models. The learning mechanisms are complementary to one another: contrastive learning shapes the embedding space across a batch of image samples; masked autoencoders focus on reconstruction of the low-frequency spatial correlations in a single image sample; and noise prediction encourages the reconstruction of the high-frequency components of an image. The combined approach results in a robust, scalable and simple-to-implement algorithm. The training process is symmetric, with 50% of patches in both views being masked at random, yielding a considerable efficiency improvement over prior contrastive learning methods. Extensive empirical studies demonstrate that CAN achieves strong downstream performance under both linear and finetuning evaluations on transfer learning and robustness tasks. CAN outperforms MAE and SimCLR when pre-training on ImageNet, but is especially useful for pre-training on larger uncurated datasets such as JFT-300M: for linear probe on ImageNet, CAN achieves 75.4% compared to 73.4% for SimCLR and 64.1% for MAE. The finetuned performance on ImageNet of our ViT-L model is 86.1%, compared to 85.5% for SimCLR, and 85.4% for MAE. The overall FLOPs load of SimCLR is 70% higher than CAN for ViT-L models.

cs.CV↗

arXiv2022

Sign and Basis Invariant Networks for Spectral Graph Representation Learning

We introduce SignNet and BasisNet -- new neural architectures that are invariant to two key symmetries displayed by eigenvectors: (i) sign flips, since if $v$ is an eigenvector then so is $-v$; and (ii) more general basis symmetries, which occur in higher dimensional eigenspaces with infinitely many choices of basis eigenvectors. We prove that under certain conditions our networks are universal, i.e., they can approximate any continuous function of eigenvectors with the desired invariances. When used with Laplacian eigenvectors, our networks are provably more expressive than existing spectral methods on graphs; for instance, they subsume all spectral graph convolutions, certain spectral graph invariants, and previously proposed graph positional encodings as special cases. Experiments show that our networks significantly outperform existing baselines on molecular graph regression, learning expressive graph representations, and learning neural fields on triangle meshes. Our code is available at https://github.com/cptq/SignNet-BasisNet .

cs.LG↗

arXiv2022

An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels

Pre-trained language models derive substantial linguistic and factual knowledge from the massive corpora on which they are trained, and prompt engineering seeks to align these models to specific tasks. Unfortunately, existing prompt engineering methods require significant amounts of labeled data, access to model parameters, or both. We introduce a new method for selecting prompt templates \textit{without labeled examples} and \textit{without direct access to the model}. Specifically, over a set of candidate templates, we choose the template that maximizes the mutual information between the input and the corresponding model output. Across 8 datasets representing 7 distinct NLP tasks, we show that when a template has high mutual information, it also has high accuracy on the task. On the largest model, selecting prompts with our method gets 90\% of the way from the average prompt accuracy to the best prompt accuracy and requires no ground truth labels.

cs.CL↗

arXiv2021

Can contrastive learning avoid shortcut solutions?

The generalization of representations learned via contrastive learning depends crucially on what features of the data are extracted. However, we observe that the contrastive loss does not always sufficiently guide which features are extracted, a behavior that can negatively impact the performance on downstream tasks via "shortcuts", i.e., by inadvertently suppressing important predictive features. We find that feature extraction is influenced by the difficulty of the so-called instance discrimination task (i.e., the task of discriminating pairs of similar points from pairs of dissimilar ones). Although harder pairs improve the representation of some features, the improvement comes at the cost of suppressing previously well represented features. In response, we propose implicit feature modification (IFM), a method for altering positive and negative samples in order to guide contrastive models towards capturing a wider variety of predictive features. Empirically, we observe that IFM reduces feature suppression, and as a result improves performance on vision and medical imaging tasks. The code is available at: \url{https://github.com/joshr17/IFM}.

cs.LG↗

arXiv2021

The Blazar Hadronic Code Comparison Project

Blazar hadronic models have been developed in the past decades as an alternative to leptonic ones. In hadronic models the gamma-ray emission is associated with synchrotron emission by protons, and/or secondary leptons produced in proton-photon interactions. Together with photons, hadronic emission models predict the emission of neutrinos that are therefore the smoking gun for acceleration of relativistic hadrons in blazar jets. The simulation of proton-photon interactions and all associated radiative processes is a complex numerical task, and different approaches to the problem have been adopted in the literature. So far, no systematic comparison between the different codes has been performed, preventing a clear understanding of the underlying uncertainties in the numerical simulations. To fill this gap, we have undertaken the first comprehensive comparison of blazar hadronic codes, and the results from this effort will be presented in this contribution.

astro-ph.HE↗

arXiv2021

A Matrix Autoencoder Framework to Align the Functional and Structural Connectivity Manifolds as Guided by Behavioral Phenotypes

We propose a novel matrix autoencoder to map functional connectomes from resting state fMRI (rs-fMRI) to structural connectomes from Diffusion Tensor Imaging (DTI), as guided by subject-level phenotypic measures. Our specialized autoencoder infers a low dimensional manifold embedding for the rs-fMRI correlation matrices that mimics a canonical outer-product decomposition. The embedding is simultaneously used to reconstruct DTI tractography matrices via a second manifold alignment decoder and to predict inter-subject phenotypic variability via an artificial neural network. We validate our framework on a dataset of 275 healthy individuals from the Human Connectome Project database and on a second clinical dataset consisting of 57 subjects with Autism Spectrum Disorder. We demonstrate that the model reliably recovers structural connectivity patterns across individuals, while robustly extracting predictive and interpretable brain biomarkers in a cross-validated setting. Finally, our framework outperforms several baselines at predicting behavioral phenotypes in both real-world datasets.

q-bio.NC↗

arXiv2021

Contrastive Learning with Hard Negative Samples

How can you sample good negative examples for contrastive learning? We argue that, as with metric learning, contrastive learning of representations benefits from hard negative samples (i.e., points that are difficult to distinguish from an anchor point). The key challenge toward using hard negatives is that contrastive methods must remain unsupervised, making it infeasible to adopt existing negative sampling strategies that use true similarity information. In response, we develop a new family of unsupervised sampling methods for selecting hard negative samples where the user can control the hardness. A limiting case of this sampling results in a representation that tightly clusters each class, and pushes different classes as far apart as possible. The proposed method improves downstream performance across multiple modalities, requires only few additional lines of code to implement, and introduces no computational overhead.

cs.LG↗

arXiv2020

Debiased Contrastive Learning

A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks. Theoretically, we establish generalization bounds for the downstream classification task.

cs.LG↗

arXiv2020

Unexpected Near-Infrared to Visible Non-linear Optical Properties from Two-Dimensional Polar Metals

Near-infrared-to-visible second harmonic generation from air-stable two-dimensional polar gallium and indium metals is described. The photonic properties of 2D metals - including the largest second-order susceptibilities reported for metals (approaching 10nm$^2$/V) - are determined by the atomic-level structure and bonding of two-to-three-atom-thick crystalline films. The bond character evolved from covalent to metallic over a few atomic layers, changing the out-of-plane metal-metal bond distances by approximately ten percent (0.2 $\unicode{x212B}$), resulting in symmetry breaking and an axial electrostatic dipole that mediated the large nonlinear response. Two different orientations of the crystalline metal atoms, corresponding to lateral displacements < 2 $\unicode{x212B}$, persisted in separate micron-scale terraces to generate distinct harmonic polarizations. This strong atomic-level structure-property interplay suggests metal photonic properties can be controlled with atomic precision.

cond-mat.mtrl-sci↗

arXiv2020

Strength from Weakness: Fast Learning Using Weak Supervision

We study generalization properties of weakly supervised learning. That is, learning where only a few "strong" labels (the actual target of our prediction) are present but many more "weak" labels are available. In particular, we show that having access to weak labels can significantly accelerate the learning rate for the strong task to the fast rate of $\mathcal{O}(\nicefrac1n)$, where $n$ denotes the number of strongly labeled data points. This acceleration can happen even if by itself the strongly labeled data admits only the slower $\mathcal{O}(\nicefrac{1}{\sqrt{n}})$ rate. The actual acceleration depends continuously on the number of weak labels available, and on the relation between the two tasks. Our theoretical results are reflected empirically across a range of tasks and illustrate how weak labels speed up learning on the strong task.

cs.LG↗

arXiv2019

Flexible Modeling of Diversity with Strongly Log-Concave Distributions

Strongly log-concave (SLC) distributions are a rich class of discrete probability distributions over subsets of some ground set. They are strictly more general than strongly Rayleigh (SR) distributions such as the well-known determinantal point process. While SR distributions offer elegant models of diversity, they lack an easy control over how they express diversity. We propose SLC as the right extension of SR that enables easier, more intuitive control over diversity, illustrating this via examples of practical importance. We develop two fundamental tools needed to apply SLC distributions to learning and inference: sampling and mode finding. For sampling we develop an MCMC sampler and give theoretical mixing time bounds. For mode finding, we establish a weak log-submodularity property for SLC functions and derive optimization guarantees for a distorted greedy algorithm.

cs.LG↗

arXiv2019

Confinement Heteroepitaxy: Realizing Atomically Thin, Half-van der Waals Materials

Three-dimensional epitaxial heterostructures are based on covalently-bonded interfaces, whereas those from 2-dimensional (2D) materials exhibit van der Waals interactions. Under the right conditions, however, material structures with mixed interfacial van der Waals and covalent bonding may be realized. Atomically thin layers formed at the epitaxial graphene (EG)/silicon carbide (SiC) interface indicate that EG/SiC interfaces provide this unique environment and enable synthesis of a rich palette of 2D materials not accessible with traditional techniques. Here, we demonstrate a method termed confinement heteroepitaxy (CHet), to realize air-stable, structurally unique, crystalline 2D-Ga, In, and Sn at the EG/SiC interface. The first intercalant layer is covalently-bonded to the SiC, and is accompanied by a vertical bonding gradient that ends with van der Waals interactions. Such structures break out of plane centrosymmetry, thereby introducing atomically thin, non-centrosymmetric 2D allotropes of 3D materials as a foundation for tunable superconductivity, topological states, and plasmonic properties.

cond-mat.mtrl-sci↗

arXiv2018

High-quality, large-grain MoS2 films grown on 100 mm sapphire substrates using a novel molybdenum precursor

Two-dimensional MoS2 is a crystalline semiconductor with high potential for numerous technologies. Research in recent years has sought to exploit the direct band gap and high carrier mobility properties of monolayer MoS2 for functional applications. To date, the production of MoS2 has remained at the research level and samples are usually synthesized in small quantities using small yield, expensive techniques. In order to realize scalable MoS2-based technology, large-area, high-quality and affordable MoS2 wafers must become available. Here we report on MoS2 films grown on 100 mm sapphire wafers by a chemical vapor deposition process utilizing hydrogen sulfide and molybdenum precursor mixtures consisting of Na2MoO4 and NaCl. The grains of these films are faceted, large-area, on the order of 1-75 microns in length on edge. Growth conditions are identified that yield monolayer MoS2 films. Raman spectroscopy shows an E2g and A1g peak separation of 17.9-19.1 cm-1. Photoluminescence spectroscopy shows that annealing the films post-growth suppresses trion emission. X-ray photoelectron spectroscopy of the annealed films shows an in increase in the S:Mo concentration ratio from 1.90:1.00 to 2.09:1.00.

cond-mat.mtrl-sci↗

arXiv2018

The race to the bottom: approaching the ideal glass?

Key to resolving the scientific challenge of the glass transition is to understand the origin of the massive increase in viscosity of liquids cooled below their melting temperature (avoiding crystallisation). A number of competing and often mutually exclusive theoretical approaches have been advanced to describe this phenomenon. Some posit a bona fide thermodynamic phase to an "ideal glass", an amorphous state with exceptionally low entropy. Other approaches are built around the concept of the glass transition as a primarily dynamic phenomenon. These fundamentally different interpretations give equally good descriptions of the data available, so it is hard to determine which -- if any -- is correct. Recently however this situation has begun to change. A consensus has emerged that one powerful means to resolve this longstanding question is to approach the putative thermodynamic transition sufficiently closely, and a number of techniques have emerged to meet this challenge. Here we review the results of some of these new techniques and discuss the implications for the existence -- or otherwise -- of the thermodynamic transition to an ideal glass.

cond-mat.soft↗