arXiv subjects

Tiberiu Tesileanu

Publications and source records attributed to Tiberiu Tesileanu.

17 recordsLinked to original sources

arXiv2024

AstroCLIP: A Cross-Modal Foundation Model for Galaxies

We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra.

astro-ph.IM↗

arXiv2023

Reusability report: Prostate cancer stratification with diverse biologically-informed neural architectures

In Elmarakeby et al., "Biologically informed deep neural network for prostate cancer discovery", a feedforward neural network with biologically informed, sparse connections (P-NET) was presented to model the state of prostate cancer. We verified the reproducibility of the study conducted by Elmarakeby et al., using both their original codebase, and our own re-implementation using more up-to-date libraries. We quantified the contribution of network sparsification by Reactome biological pathways, and confirmed its importance to P-NET's superior performance. Furthermore, we explored alternative neural architectures and approaches to incorporating biological information into the networks. We experimented with three types of graph neural networks on the same training data, and investigated the clinical prediction agreement between different models. Our analyses demonstrated that deep neural networks with distinct architectures make incorrect predictions for individual patient that are persistent across different initializations of a specific neural architecture. This suggests that different neural architectures are sensitive to different aspects of the data, an important yet under-explored challenge for clinical prediction tasks.

cs.LG↗

arXiv2023

xVal: A Continuous Numerical Tokenization for Scientific Language Models

Due in part to their discontinuous and discrete default encodings for numbers, Large Language Models (LLMs) have not yet been commonly used to process numerically-dense scientific datasets. Rendering datasets as text, however, could help aggregate diverse and multi-modal scientific data into a single training corpus, thereby potentially facilitating the development of foundation models for science. In this work, we introduce xVal, a strategy for continuously tokenizing numbers within language models that results in a more appropriate inductive bias for scientific applications. By training specially-modified language models from scratch on a variety of scientific datasets formatted as text, we find that xVal generally outperforms other common numerical tokenization strategies on metrics including out-of-distribution generalization and computational efficiency.

stat.ML↗

arXiv2023

Multiple Physics Pretraining for Physical Surrogate Models

We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling of spatiotemporal systems with transformers. In MPP, rather than training one model on a specific physical system, we train a backbone model to predict the dynamics of multiple heterogeneous physical systems simultaneously in order to learn features that are broadly useful across systems and facilitate transfer. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks over a broad fluid mechanics-oriented benchmark. We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on all pretraining sub-tasks without the need for finetuning. For downstream tasks, we demonstrate that finetuning MPP-trained models results in more accurate predictions across multiple time-steps on systems with previously unseen physical components or higher dimensional systems compared to training from scratch or finetuning pretrained video foundation models. We open-source our code and model weights trained at multiple scales for reproducibility.

cs.LG↗

arXiv2023

Constrained Predictive Coding as a Biologically Plausible Model of the Cortical Hierarchy

Predictive coding has emerged as an influential normative model of neural computation, with numerous extensions and applications. As such, much effort has been put into mapping PC faithfully onto the cortex, but there are issues that remain unresolved or controversial. In particular, current implementations often involve separate value and error neurons and require symmetric forward and backward weights across different brain regions. These features have not been experimentally confirmed. In this work, we show that the PC framework in the linear regime can be modified to map faithfully onto the cortical hierarchy in a manner compatible with empirical observations. By employing a disentangling-inspired constraint on hidden-layer neural activities, we derive an upper bound for the PC objective. Optimization of this upper bound leads to an algorithm that shows the same performance as the original objective and maps onto a biologically plausible network. The units of this network can be interpreted as multi-compartmental neurons with non-Hebbian learning rules, with a remarkable resemblance to recent experimental findings. There exist prior models which also capture these features, but they are phenomenological, while our work is a normative derivation. The network we derive does not involve one-to-one connectivity or signal multiplexing, which the phenomenological models required, indicating that these features are not necessary for learning in the cortex. The normative nature of our algorithm in the simplified linear case allows us to prove interesting properties of the framework and analytically understand the computational role of our network's components. The parameters of our network have natural interpretations as physiological quantities in a multi-compartmental model of pyramidal neurons, providing a concrete link between PC and experimental measurements carried out in the cortex.

q-bio.NC↗

arXiv2022

An online algorithm for contrastive Principal Component Analysis

Finding informative low-dimensional representations that can be computed efficiently in large datasets is an important problem in data analysis. Recently, contrastive Principal Component Analysis (cPCA) was proposed as a more informative generalization of PCA that takes advantage of contrastive learning. However, the performance of cPCA is sensitive to hyper-parameter choice and there is currently no online algorithm for implementing cPCA. Here, we introduce a modified cPCA method, which we denote cPCA*, that is more interpretable and less sensitive to the choice of hyper-parameter. We derive an online algorithm for cPCA* and show that it maps onto a neural network with local learning rules, so it can potentially be implemented in energy efficient neuromorphic hardware. We evaluate the performance of our online algorithm on real datasets and highlight the differences and similarities with the original formulation.

stat.ML↗

arXiv2021

Neural circuits for dynamics-based segmentation of time series

The brain must extract behaviorally relevant latent variables from the signals streamed by the sensory organs. Such latent variables are often encoded in the dynamics that generated the signal rather than in the specific realization of the waveform. Therefore, one problem faced by the brain is to segment time series based on underlying dynamics. We present two algorithms for performing this segmentation task that are biologically plausible, which we define as acting in a streaming setting and all learning rules being local. One algorithm is model-based and can be derived from an optimization problem involving a mixture of autoregressive processes. This algorithm relies on feedback in the form of a prediction error, and can also be used for forecasting future samples. In some brain regions, such as the retina, the feedback connections necessary to use the prediction error for learning are absent. For this case, we propose a second, model-free algorithm that uses a running estimate of the autocorrelation structure of the signal to perform the segmentation. We show that both algorithms do well when tasked with segmenting signals drawn from autoregressive models with piecewise-constant parameters. In particular, the segmentation accuracy is similar to that obtained from oracle-like methods in which the ground-truth parameters of the autoregressive models are known. We also test our methods on datasets generated by alternating snippets of voice recordings. We provide implementations of our algorithms at https://github.com/ttesileanu/bio-time-series.

q-bio.NC↗

arXiv2020

Efficient coding of natural scene statistics predicts discrimination thresholds for grayscale textures

Previously, in (Hermundstad et al., 2014), we showed that when sampling is limiting, the efficient coding principle leads to a "variance is salience" hypothesis, and that this hypothesis accounts for visual sensitivity to binary image statistics. Here, using extensive new psychophysical data and image analysis, we show that this hypothesis accounts for visual sensitivity to a large set of grayscale image statistics at a striking level of detail, and also identify the limits of the prediction. We define a 66-dimensional space of local grayscale light-intensity correlations, and measure the relevance of each direction to natural scenes. The "variance is salience" hypothesis predicts that two-point correlations are most salient, and predicts their relative salience. We tested these predictions in a texture-segregation task using un-natural, synthetic textures. As predicted, correlations beyond second order are not salient, and predicted thresholds for over 300 second-order correlations match psychophysical thresholds closely (median fractional error <0.13).

q-bio.NC↗

arXiv2019

Adaptation of olfactory receptor abundances for efficient coding

Olfactory receptor usage is highly heterogeneous, with some receptor types being orders of magnitude more abundant than others. We propose an explanation for this striking fact: the receptor distribution is tuned to maximally represent information about the olfactory environment in a regime of efficient coding that is sensitive to the global context of correlated sensor responses. This model predicts that in mammals, where olfactory sensory neurons are replaced regularly, receptor abundances should continuously adapt to odor statistics. Experimentally, increased exposure to odorants leads variously, but reproducibly, to increased, decreased, or unchanged abundances of different activated receptors. We demonstrate that this diversity of effects is required for efficient coding when sensors are broadly correlated, and provide an algorithm for predicting which olfactory receptors should increase or decrease in abundance following specific environmental changes. Finally, we give simple dynamical rules for neural birth and death processes that might underlie this adaptation.

q-bio.NC↗

arXiv2017

Dynamics of adaptive immunity against phage in bacterial populations

The CRISPR (clustered regularly interspaced short palindromic repeats) mechanism allows bacteria to adaptively defend against phages by acquiring short genomic sequences (spacers) that target specific sequences in the viral genome. We propose a population dynamical model where immunity can be both acquired and lost. The model predicts regimes where bacterial and phage populations can co-exist, others where the populations exhibit damped oscillations, and still others where one population is driven to extinction. Our model considers two key parameters: (1) ease of acquisition and (2) spacer effectiveness in conferring immunity. Analytical calculations and numerical simulations show that if spacers differ mainly in ease of acquisition, or if the probability of acquiring them is sufficiently high, bacteria develop a diverse population of spacers. On the other hand, if spacers differ mainly in their effectiveness, their final distribution will be highly peaked, akin to a "winner-take-all" scenario, leading to a specialized spacer distribution. Bacteria can interpolate between these limiting behaviors by actively tuning their overall acquisition probability.

physics.bio-ph↗

arXiv2017

Rules and mechanisms for efficient two-stage learning in neural circuits

Trial-and-error learning requires evaluating variable actions and reinforcing successful variants. In songbirds, vocal exploration is induced by LMAN, the output of a basal ganglia-circuit that also contributes a corrective bias to the vocal output. This bias is gradually consolidated in RA, a motor cortex analogue downstream of LMAN. We develop a new model of such two-stage learning. Using stochastic gradient descent, we derive how the activity in 'tutor' circuits (e.g., LMAN) should match plasticity mechanisms in 'student' circuits (e.g., RA) to achieve efficient learning. We further describe a reinforcement learning framework through which the tutor can build its teaching signal. We show that mismatches between the tutor signal and the plasticity mechanism can impair learning. Applied to birdsong, our results predict the temporal structure of the corrective bias from LMAN given a plasticity rule in RA. Our framework can be applied predictively to other paired brain areas showing two-stage learning.

q-bio.NC↗

arXiv2014

Protein sectors: statistical coupling analysis versus conservation

Statistical coupling analysis (SCA) is a method for analyzing multiple sequence alignments that was used to identify groups of coevolving residues termed "sectors". The method applies spectral analysis to a matrix obtained by combining correlation information with sequence conservation. It has been asserted that the protein sectors identified by SCA are functionally significant, with different sectors controlling different biochemical properties of the protein. Here we reconsider the available experimental data and note that it involves almost exclusively proteins with a single sector. We show that in this case sequence conservation is the dominating factor in SCA, and can alone be used to make statistically equivalent functional predictions. Therefore, we suggest shifting the experimental focus to proteins for which SCA identifies several sectors. Correlations in protein alignments, which have been shown to be informative in a number of independent studies, would then be less dominated by sequence conservation.

q-bio.BM↗

arXiv2011

Multi-Matrix Models and Tri-Sasaki Einstein Spaces

Localization methods reduce the path integrals in {\cal N} >= 2 supersymmetric Chern-Simons gauge theories on S^3 to multi-matrix integrals. A recent evaluation of such a two-matrix integral for the {\cal N}=6 superconformal U(N) x U(N) ABJM theory produced detailed agreement with the AdS/CFT correspondence, explaining, in particular the N^{3/2} scaling of the free energy. We study a class of p-matrix integrals describing {\cal N}=3 superconformal U(N)^p Chern-Simons gauge theories. We present a simple method that allows us to evaluate the eigenvalue densities and the free energies in the large N limit keeping the Chern-Simons levels k_i fixed. The dual M-theory backgrounds are AdS_4 x Y, where Y are seven-dimensional tri-Sasaki Einstein spaces specified by the k_i. The gravitational free energy scales inversely with the square root of the volume of Y. We find a general formula for the p-matrix free energies that agrees with the available results for volumes of the tri-Sasaki Einstein spaces Y, thus providing a thorough test of the corresponding AdS_4/CFT_3 dualities. This formula is consistent with the Seiberg duality conjectured for Chern-Simons gauge theories.

hep-th↗

arXiv2010

Emergent Quantum Near-Criticality from Baryonic Black Branes

We find new black 3-brane solutions describing the "conifold gauge theory" at nonzero temperature and baryonic chemical potential. Of particular interest is the low-temperature limit where we find a new kind of weakly curved near-horizon geometry; it is a warped product AdS_2 x R^3 x T^{1,1} with warp factors that are powers of the logarithm of the AdS radius. Thus, our solution encodes a new type of emergent quantum near-criticality. We carry out some stability checks for our solutions. We also set up a consistent ansatz for baryonic black 2-branes of M-theory that are asymptotic to AdS_4 x Q^{1,1,1}.

hep-th↗

arXiv2010

Membranes with Topological Charge and AdS4/CFT3 Correspondence

If the second Betti number b_2 of a Sasaki-Einstein manifold Y^7 does not vanish, then M-theory on AdS_4 x Y^7 possesses "topological" U(1)^{b_2} gauge symmetry. The corresponding Abelian gauge fields come from three-form fluctuations with one index in AdS_4 and the other two in Y^7. We find black membrane solutions carrying one of these U(1) charges. In the zero temperature limit, our solutions interpolate between AdS_4 x Y^7 in the UV and AdS_2 x R^2 x squashed Y^7 in the IR. In fact, the AdS_2 x R^2 x squashed Y^7 background is by itself a solution of the supergravity equations of motion. These solutions do not appear to preserve any supersymmetry. We search for their possible instabilities and do not find any. We also discuss the meaning of our charged membrane backgrounds in a dual quiver Chern-Simons gauge theory with a global U(1) charge density. Finally, we present a simple analytic solution which has the same IR but different UV behavior. We reduce this solution to type IIA string theory, and perform T-duality to type IIB. The type IIB metric turns out to be a product of the squashed Y^7 and the extremal BTZ black hole. We discuss an interpretation of this type IIB background in terms of the (1+1)-dimensional CFT on D3-branes partially wrapped over the squashed Y^7.

hep-th↗

arXiv2009

Superconductors from Superstrings

We establish that in a large class of strongly coupled 3+1 dimensional N=1 quiver conformal field theories with gravity duals, adding a chemical potential for the R-charge leads to the existence of superfluid states in which a chiral primary operator of the schematic form O = λλ+ W condenses. Here λis a gluino and W is the superpotential. Our argument is based on the construction of a consistent truncation of type IIB supergravity that includes a U(1) gauge field and a complex scalar.

hep-th↗

arXiv2006

CMB statistical anisotropy, multipole vectors and the influence of the dipole

A simple algorithm which gives the multipole vectors in terms of the roots of a polynomial is given. We find that the reported alignmet of the low l multipole vectors can be summarised as an anti-alignmet of these with the dipole direction. This anti-alignment is not only present in l=2 and 3 but also for l=5 and higher. This alignment is likely due to non-linearity in the data processing. Our results are based on the three year WMAP data, we also list corresponding results for the first year data.

astro-ph↗