arXiv subjects

Lu Yin

Publications and source records attributed to Lu Yin.

At least 109 records · Page 6Linked to original sources

arXiv2023

Are Large Kernels Better Teachers than Transformers for ConvNets?

This paper reveals a new appeal of the recently emerged large-kernel Convolutional Neural Networks (ConvNets): as the teacher in Knowledge Distillation (KD) for small-kernel ConvNets. While Transformers have led state-of-the-art (SOTA) performance in various fields with ever-larger models and labeled data, small-kernel ConvNets are considered more suitable for resource-limited applications due to the efficient convolution operation and compact weight sharing. KD is widely used to boost the performance of small-kernel ConvNets. However, previous research shows that it is not quite effective to distill knowledge (e.g., global information) from Transformers to small-kernel ConvNets, presumably due to their disparate architectures. We hereby carry out a first-of-its-kind study unveiling that modern large-kernel ConvNets, a compelling competitor to Vision Transformers, are remarkably more effective teachers for small-kernel ConvNets, due to more similar architectures. Our findings are backed up by extensive experiments on both logit-level and feature-level KD ``out of the box", with no dedicated architectural nor training recipe modifications. Notably, we obtain the \textbf{best-ever pure ConvNet} under 30M parameters with \textbf{83.1\%} top-1 accuracy on ImageNet, outperforming current SOTA methods including ConvNeXt V2 and Swin V2. We also find that beneficial characteristics of large-kernel ConvNets, e.g., larger effective receptive fields, can be seamlessly transferred to students through this large-to-small kernel distillation. Code is available at: \url{https://github.com/VITA-Group/SLaK}.

cs.CV↗

arXiv2023

Is Cosmic Birefringence model-dependent?

Exciting clues to isotropic cosmic birefringence have recently been detected in the $EB$ cross-power spectra of the polarization data of the cosmic microwave background (CMB). Early Dark Energy (EDE) models with a pseudoscalar field coupled to photons via a Chern-Simons term can be used to explain this phenomenon, and can also potentially be used to simultaneously resolve the $H_0$ tension. In this work we incorporate an early dark energy scalar field, including a Chern-Simons coupling, into an existing Boltzmann solver and numerically recover the $EB$ cross-power spectrum for two models in the literature; the $α$-attractor, and the Rock `n' Roll field. We find that both the models fit the $EB$ spectra, and the $EB$ spectra alone do not possess sufficient constraining power to distinguish the two models based on current data.

astro-ph.CO↗

arXiv2023

Lottery Pools: Winning More by Interpolating Tickets without Increasing Training or Inference Cost

Lottery tickets (LTs) is able to discover accurate and sparse subnetworks that could be trained in isolation to match the performance of dense networks. Ensemble, in parallel, is one of the oldest time-proven tricks in machine learning to improve performance by combining the output of multiple independent models. However, the benefits of ensemble in the context of LTs will be diluted since ensemble does not directly lead to stronger sparse subnetworks, but leverages their predictions for a better decision. In this work, we first observe that directly averaging the weights of the adjacent learned subnetworks significantly boosts the performance of LTs. Encouraged by this observation, we further propose an alternative way to perform an 'ensemble' over the subnetworks identified by iterative magnitude pruning via a simple interpolating strategy. We call our method Lottery Pools. In contrast to the naive ensemble which brings no performance gains to each single subnetwork, Lottery Pools yields much stronger sparse subnetworks than the original LTs without requiring any extra training or inference cost. Across various modern architectures on CIFAR-10/100 and ImageNet, we show that our method achieves significant performance gains in both, in-distribution and out-of-distribution scenarios. Impressively, evaluated with VGG-16 and ResNet-18, the produced sparse subnetworks outperform the original LTs by up to 1.88% on CIFAR-100 and 2.36% on CIFAR-100-C; the resulting dense network surpasses the pre-trained dense-model up to 2.22% on CIFAR-100 and 2.38% on CIFAR-100-C.

cs.LG↗

arXiv2023

Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks

Feature selection that selects an informative subset of variables from data not only enhances the model interpretability and performance but also alleviates the resource demands. Recently, there has been growing attention on feature selection using neural networks. However, existing methods usually suffer from high computational costs when applied to high-dimensional datasets. In this paper, inspired by evolution processes, we propose a novel resource-efficient supervised feature selection method using sparse neural networks, named \enquote{NeuroFS}. By gradually pruning the uninformative features from the input layer of a sparse neural network trained from scratch, NeuroFS derives an informative subset of features efficiently. By performing several experiments on $11$ low and high-dimensional real-world benchmarks of different types, we demonstrate that NeuroFS achieves the highest ranking-based score among the considered state-of-the-art supervised feature selection models. The code is available on GitHub.

cs.NE↗

arXiv2023

Is the Observable Universe Consistent with the Cosmological Principle?

The Cosmological Principle (CP) -- the notion that the Universe is spatially isotropic and homogeneous on large scales -- underlies a century of progress in cosmology. It is conventionally formulated through the Friedmann-Lemaître-Robertson-Walker (FLRW) cosmologies as the spacetime metric, and culminates in the successful and highly predictive $Λ$-Cold-Dark-Matter ($Λ$CDM) model. Yet, tensions have emerged within the $Λ$CDM model, most notably a statistically significant discrepancy in the value of the Hubble constant, $H_0$. Since the notion of cosmic expansion determined by a single parameter is intimately tied to the CP, implications of the $H_0$ tension may extend beyond $Λ$CDM to the CP itself. This review surveys current observational hints for deviations from the expectations of the CP, highlighting synergies and disagreements that warrant further study. Setting aside the debate about individual large structures, potential deviations from the CP include variations of cosmological parameters on the sky, discrepancies in the cosmic dipoles, and mysterious alignments in quasar polarizations and galaxy spins. While it is possible that a host of observational systematics are impacting results, it is equally plausible that precision cosmology may have outgrown the FLRW paradigm, an extremely pragmatic but non-fundamental symmetry assumption.

astro-ph.CO↗

arXiv2022

Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training

Recent works on sparse neural network training (sparse training) have shown that a compelling trade-off between performance and efficiency can be achieved by training intrinsically sparse neural networks from scratch. Existing sparse training methods usually strive to find the best sparse subnetwork possible in one single run, without involving any expensive dense or pre-training steps. For instance, dynamic sparse training (DST), is capable of reaching a competitive performance of dense training by iteratively evolving the sparse topology during the course of training. In this paper, we argue that it is better to allocate the limited resources to create multiple low-loss sparse subnetworks and superpose them into a stronger one, instead of allocating all resources entirely to find an individual subnetwork. To achieve this, two desiderata are required: (1) efficiently producing many low-loss subnetworks, the so-called cheap tickets, within one training process limited to the standard training time used in dense training; (2) effectively superposing these cheap tickets into one stronger subnetwork. To corroborate our conjecture, we present a novel sparse training approach, termed Sup-tickets, which can satisfy the above two desiderata concurrently in a single sparse-to-sparse training process. Across various modern architectures on CIFAR-10/100 and ImageNet, we show that Sup-tickets integrates seamlessly with the existing sparse training methods and demonstrates consistent performance improvement.

cs.LG↗

arXiv2022

Gravitational waves from the vacuum decay with LISA

We investigate the gravitational wave spectrum resulted from the cosmological first-order phase transition. We compare two models; one is a scalar field model without gravitation, while the other is a scalar field model with gravitation. Based on the sensitivity curves of the LISA space-based interferometer on the stochastic gravitational-wave background, we compare the difference between the gravitational wave spectra of the former and the latter cases resulted from the bubble collision process. Especially, we calculated the speed of the bubble wall before collision for the two models numerically. We show that the difference between the amplitudes of those spectra can clearly distinguish between the two models. We expect that the LISA with Signal to Noise Ratio =10 could observe the spectrum as the fast first-order phase transition.

gr-qc↗

arXiv2022

Reducing the $H_0$ Tension with Exponential Acoustic Dark Energy

The Hubble tension arises from different observations between the late-time and early Universe. We explore a new model with dark fluid, called the exponential Acoustic Dark Energy (eADE) model, to relieve the Hubble tension. The eADE model gives an exponential form of the equation of state (EoS) in the acoustic dark energy, which is the first time to explore an exponential form for the EoS. In this model, the gravitational effects from the acoustic oscillations of the model can impact the CMB phenomena at the matter radiation equally epoch. We give the constraints of the eADE model by the current cosmological dataset. The comparison of the phenomena with the standard model can be shown through CMB and matter power spectra. The fitting results of our model have $H_0 = 70.06^{+1.13}_{-1.09}$ in 95$\%$ C.L. and a smaller best-fit value than $Λ$CDM.

astro-ph.CO↗

arXiv2022

On Larger $H_0$ Values in the CMB Dipole Direction

On the assumption that quasars (QSO) and gamma-ray bursts (GRB) represent \textit{standardisable candles}, we provide evidence that the Hubble constant $H_0$ adopts larger values in hemispheres aligned with the CMB dipole direction. If substantiated, this trend signals a departure from FLRW cosmology. In particular, QSOs show a definite trend, whereas our findings in GRBs are consistent with an isotropic Universe, but we show in a sample of GRBs calibrated with Type Ia supernovae (SN) that this conclusion may change as one focuses on GRBs more closely (mis)aligned with the CMB dipole direction. The statistical significance in QSOs alone is $\gtrsim 2 σ$ and when combined with similar trends in strong lensing, Type Ia SN and calibrated GRBs, this increases to $\sim 3 σ$. Our findings are consistent with reported discrepancies in the cosmic dipole and anisotropies in galaxy cluster scaling relations. The reported variations in $H_0$ across the sky suggest that Hubble tension may be a symptom of a deeper cosmological malaise.

astro-ph.CO↗

arXiv2022

Semantic-Based Few-Shot Learning by Interactive Psychometric Testing

Few-shot classification tasks aim to classify images in query sets based on only a few labeled examples in support sets. Most studies usually assume that each image in a task has a single and unique class association. Under these assumptions, these algorithms may not be able to identify the proper class assignment when there is no exact matching between support and query classes. For example, given a few images of lions, bikes, and apples to classify a tiger. However, in a more general setting, we could consider the higher-level concept, the large carnivores, to match the tiger to the lion for semantic classification. Existing studies rarely considered this situation due to the incompatibility of label-based supervision with complex conception relationships. In this work, we advance the few-shot learning towards this more challenging scenario, the semantic-based few-shot learning, and propose a method to address the paradigm by capturing the inner semantic relationships using interactive psychometric learning. The experiment results on the CIFAR-100 dataset show the superiority of our method for the semantic-based few-shot learning compared to the baseline.

cs.CV↗

arXiv2022

Hints of FLRW Breakdown from Supernovae

A 10\% difference in the scale for the Hubble parameter constitutes a clear problem for cosmology. Here, considering angular distribution of Type Ia supernovae (SN) within the Pantheon compilation and working within flat $Λ$CDM cosmology, we observe a correlation between higher $H_0$ and the CMB dipole direction, confirming our previous results for strongly-lensed quasars \cite{Krishnan:2021dyb}. Concretely, we record a $\sim 1$ km/s/Mpc variation in $H_0$ at antipodal points on the sky within the Pantheon sample, which is evident in the Low $z$ subsample ($z \lesssim 0.075$) and gets enhanced by higher redshift SN. Our work raises the possibility that we may be at the precision required to probe anisotropic Hubble expansions, while providing a concrete prediction for future inferences of $H_0$.

astro-ph.CO↗

arXiv2022

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization). The former method suffers from an extremely large computation cost and the latter usually struggles with insufficient performance. In comparison, during-training pruning, a class of pruning methods that simultaneously enjoys the training/inference efficiency and the comparable performance, temporarily, has been less explored. To better understand during-training pruning, we quantitatively study the effect of pruning throughout training from the perspective of pruning plasticity (the ability of the pruned networks to recover the original performance). Pruning plasticity can help explain several other empirical observations about neural network pruning in literature. We further find that pruning plasticity can be substantially improved by injecting a brain-inspired mechanism called neuroregeneration, i.e., to regenerate the same number of connections as pruned. We design a novel gradual magnitude pruning (GMP) method, named gradual pruning with zero-cost neuroregeneration (\textbf{GraNet}), that advances state of the art. Perhaps most impressively, its sparse-to-sparse version for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods with ResNet-50 on ImageNet without extending the training time. We release all codes in https://github.com/Shiweiliuiiiiiii/GraNet.

cs.LG↗

arXiv2021

Hierarchical Semantic Segmentation using Psychometric Learning

Assigning meaning to parts of image data is the goal of semantic image segmentation. Machine learning methods, specifically supervised learning is commonly used in a variety of tasks formulated as semantic segmentation. One of the major challenges in the supervised learning approaches is expressing and collecting the rich knowledge that experts have with respect to the meaning present in the image data. Towards this, typically a fixed set of labels is specified and experts are tasked with annotating the pixels, patches or segments in the images with the given labels. In general, however, the set of classes does not fully capture the rich semantic information present in the images. For example, in medical imaging such as histology images, the different parts of cells could be grouped and sub-grouped based on the expertise of the pathologist. To achieve such a precise semantic representation of the concepts in the image, we need access to the full depth of knowledge of the annotator. In this work, we develop a novel approach to collect segmentation annotations from experts based on psychometric testing. Our method consists of the psychometric testing procedure, active query selection, query enhancement, and a deep metric learning model to achieve a patch-level image embedding that allows for semantic segmentation of images. We show the merits of our method with evaluation on the synthetically generated image, aerial image and histology image.

cs.CV↗

arXiv2021

Does Hubble Tension Signal a Breakdown in FLRW Cosmology?

The tension between early and late Universe probes of the Hubble constant has motivated various new FLRW cosmologies. Here, we reanalyse the Hubble tension with a recent age of the Universe constraint. This allows us to restrict attention to matter and a dark energy sector that we treat without assuming a specific model. Assuming analyticity of the Hubble parameter $H(z)$, and a generic low redshift modification to flat $Λ$CDM, we find that low redshift data ($z \lesssim 2.5$) and well-motivated priors only permit a dark energy sector close to the cosmological constant $Λ$. This restriction rules out late Universe modifications within FLRW. We show that early Universe physics that alters the sound horizon can yield an upper limit of $H_0 \sim 71 \pm 1$ km/s/Mpc. Since various local determinations may be converging to $H_0 \sim 73$ km/s/Mpc, a breakdown of the FLRW framework is a plausible resolution. We outline how future data, in particular strongly lensed quasar data, could also provide further confirmations of such a resolution.

astro-ph.CO↗

arXiv2021

Can dark energy be dynamical?

We highlight shortcomings of the dynamical dark energy (DDE) paradigm. For parametric models with equation of state (EOS), $w(z) = w_0 + w_a f(z)$ for a given function of redshift $f(z)$, we show that the errors in $w_a$ are sensitive to $f(z)$: if $f(z)$ increases quickly with redshift $z$, then errors in $w_a$ are smaller, and vice versa. As a result, parametric DDE models suffer from a degree of arbitrariness and focusing too much on one model runs the risk that DDE may be overlooked. In particular, we show the ubiquitous Chevallier-Polarski-Linder model is one of the least sensitive to DDE. We also comment on ``wiggles" in $w(z)$ uncovered in non-parametric reconstructions. Concretely, we isolate the most relevant Fourier modes in the wiggles, model them and fit them back to the original data to confirm the wiggles at $\lesssim2σ$. We delve into the assumptions going into the reconstruction and argue that the assumed correlations, which clearly influence the wiggles, place strong constraints on field theory models of DDE.

astro-ph.CO↗

arXiv2021

Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training

In this paper, we introduce a new perspective on training deep neural networks capable of state-of-the-art performance without the need for the expensive over-parameterization by proposing the concept of In-Time Over-Parameterization (ITOP) in sparse training. By starting from a random sparse network and continuously exploring sparse connectivities during training, we can perform an Over-Parameterization in the space-time manifold, closing the gap in the expressibility between sparse training and dense training. We further use ITOP to understand the underlying mechanism of Dynamic Sparse Training (DST) and indicate that the benefits of DST come from its ability to consider across time all possible parameters when searching for the optimal sparse connectivity. As long as there are sufficient parameters that have been reliably explored during training, DST can outperform the dense neural network by a large margin. We present a series of experiments to support our conjecture and achieve the state-of-the-art sparse training performance with ResNet-50 on ImageNet. More impressively, our method achieves dominant performance over the overparameterization-based sparse methods at extreme sparsity levels. When trained on CIFAR-100, our method can match the performance of the dense model even at an extreme sparsity (98%). Code can be found https://github.com/Shiweiliuiiiiiii/In-Time-Over-Parameterization.

cs.LG↗

arXiv2021

Rethinking Lifelong Sequential Recommendation with Incremental Multi-Interest Attention

Sequential recommendation plays an increasingly important role in many e-commerce services such as display advertisement and online shopping. With the rapid development of these services in the last two decades, users have accumulated a massive amount of behavior data. Richer sequential behavior data has been proven to be of great value for sequential recommendation. However, traditional sequential models fail to handle users' lifelong sequences, as their linear computational and storage cost prohibits them from performing online inference. Recently, lifelong sequential modeling methods that borrow the idea of memory networks from NLP are proposed to address this issue. However, the RNN-based memory networks built upon intrinsically suffer from the inability to capture long-term dependencies and may instead be overwhelmed by the noise on extremely long behavior sequences. In addition, as the user's behavior sequence gets longer, more interests would be demonstrated in it. It is therefore crucial to model and capture the diverse interests of users. In order to tackle these issues, we propose a novel lifelong incremental multi-interest self attention based sequential recommendation model, namely LimaRec. Our proposed method benefits from the carefully designed self-attention to identify relevant information from users' behavior sequences with different interests. It is still able to incrementally update users' representations for online inference, similarly to memory network based approaches. We extensively evaluate our method on four real-world datasets and demonstrate its superior performances compared to the state-of-the-art baselines.

cs.IR↗

arXiv2021

Linear-Time Self Attention with Codeword Histogram for Efficient Recommendation

Self-attention has become increasingly popular in a variety of sequence modeling tasks from natural language processing to recommendation, due to its effectiveness. However, self-attention suffers from quadratic computational and memory complexities, prohibiting its applications on long sequences. Existing approaches that address this issue mainly rely on a sparse attention context, either using a local window, or a permuted bucket obtained by locality-sensitive hashing (LSH) or sorting, while crucial information may be lost. Inspired by the idea of vector quantization that uses cluster centroids to approximate items, we propose LISA (LInear-time Self Attention), which enjoys both the effectiveness of vanilla self-attention and the efficiency of sparse attention. LISA scales linearly with the sequence length, while enabling full contextual attention via computing differentiable histograms of codeword distributions. Meanwhile, unlike some efficient attention methods, our method poses no restriction on casual masking or sequence length. We evaluate our method on four real-world datasets for sequential recommendation. The results show that LISA outperforms the state-of-the-art efficient attention methods in both performance and speed; and it is up to 57x faster and 78x more memory efficient than vanilla self-attention.

cs.IR↗