arXiv subjects

Chen Sun

Publications and source records attributed to Chen Sun.

At least 163 records · Page 9Linked to original sources

arXiv2022

Attention Bottlenecks for Multimodal Fusion

Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (`late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses `fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.

cs.CV↗

arXiv2022

Dark Matter In Extreme Astrophysical Environments

Exploring dark matter via observations of extreme astrophysical environments -- defined here as heavy compact objects such as white dwarfs, neutron stars, and black holes, as well as supernovae and compact object merger events -- has been a major field of growth since the last Snowmass process. Theoretical work has highlighted the utility of current and near-future observatories to constrain novel dark matter parameter space across the full mass range. This includes gravitational wave instruments and observatories spanning the electromagnetic spectrum, from radio to gamma-rays. While recent searches already provide leading sensitivity to various dark matter models, this work also highlights the need for theoretical astrophysics research to better constrain the properties of these extreme astrophysical systems. The unique potential of these search signatures to probe dark matter adds motivation to proposed next-generation astronomical and gravitational wave instruments.

hep-ph↗

arXiv2022

A Novel Non-Volatile Inverter-based CiM: Continuous Sign Weight Transition and Low Power on-Chip Training

In this work, we report a novel design, one-transistor-one-inverter (1T1I), to satisfy high speed and low power on-chip training requirements. By leveraging doped HfO2 with ferroelectricity, a non-volatile inverter is successfully demonstrated, enabling desired continuous weight transition between negative and positive via the programmable threshold voltage (VTH) of ferroelectric field-effect transistors (FeFETs). Compared with commonly used designs with the similar function, 1T1I uniquely achieves pure on-chip-based weight transition at an optimized working current without relying on assistance from off-chip calculation units for signed-weight comparison, facilitating high-speed training at low power consumption. Further improvements in linearity and training speed can be obtained via a two-transistor-one-inverter (2T1I) design. Overall, focusing on energy and time efficiencies, this work provides a valuable design strategy for future FeFET-based computing-in-memory (CiM).

cond-mat.mes-hall↗

arXiv2022

No-go rules for multitime Landau-Zener models

Multitime Landau-Zener (MTLZ) model is a class of exactly solvable quantum many-body models which is multitstate and multitime generalization of the two-state Landau-Zener model. Currently discovered MTLZ models include the "hypercubes", the "fans" and their direct product models. In this work, we prove two no-go rules, named the "no $K_{3,3}$" rule and the "no $1221$" rule, which forbid the existence of exact solutions for models with certain structures of interactions. We further apply these rules to show that for models with no more than $9$ states, besides the models mentioned above there are no other MTLZ models. We also propose a scheme to systematically classify cases that could possibly host MTLZ models. Our work could serve as a guideline to search for new exactly solvable models within the MTLZ class.

quant-ph↗

arXiv2022

A New Knowledge Distillation Network for Incremental Few-Shot Surface Defect Detection

Surface defect detection is one of the most essential processes for industrial quality inspection. Deep learning-based surface defect detection methods have shown great potential. However, the well-performed models usually require large training data and can only detect defects that appeared in the training stage. When facing incremental few-shot data, defect detection models inevitably suffer from catastrophic forgetting and misclassification problem. To solve these problems, this paper proposes a new knowledge distillation network, called Dual Knowledge Align Network (DKAN). The proposed DKAN method follows a pretraining-finetuning transfer learning paradigm and a knowledge distillation framework is designed for fine-tuning. Specifically, an Incremental RCNN is proposed to achieve decoupled stable feature representation of different categories. Under this framework, a Feature Knowledge Align (FKA) loss is designed between class-agnostic feature maps to deal with catastrophic forgetting problems, and a Logit Knowledge Align (LKA) loss is deployed between logit distributions to tackle misclassification problems. Experiments have been conducted on the incremental Few-shot NEU-DET dataset and results show that DKAN outperforms other methods on various few-shot scenes, up to 6.65% on the mean Average Precision metric, which proves the effectiveness of the proposed method.

cs.CV↗

arXiv2022

TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency

YouTube users looking for instructions for a specific task may spend a long time browsing content trying to find the right video that matches their needs. Creating a visual summary (abridged version of a video) provides viewers with a quick overview and massively reduces search time. In this work, we focus on summarizing instructional videos, an under-explored area of video summarization. In comparison to generic videos, instructional videos can be parsed into semantically meaningful segments that correspond to important steps of the demonstrated task. Existing video summarization datasets rely on manual frame-level annotations, making them subjective and limited in size. To overcome this, we first automatically generate pseudo summaries for a corpus of instructional videos by exploiting two key assumptions: (i) relevant steps are likely to appear in multiple videos of the same task (Task Relevance), and (ii) they are more likely to be described by the demonstrator verbally (Cross-Modal Saliency). We propose an instructional video summarization network that combines a context-aware temporal video encoder and a segment scoring transformer. Using pseudo summaries as weak supervision, our network constructs a visual summary for an instructional video given only video and transcribed speech. To evaluate our model, we collect a high-quality test set, WikiHow Summaries, by scraping WikiHow articles that contain video demonstrations and visual depictions of steps allowing us to obtain the ground-truth summaries. We outperform several baselines and a state-of-the-art video summarization model on this new benchmark.

cs.CV↗

arXiv2022

Beyond Transfer Learning: Co-finetuning for Action Localisation

Transfer learning is the predominant paradigm for training deep networks on small target datasets. Models are typically pretrained on large ``upstream'' datasets for classification, as such labels are easy to collect, and then finetuned on ``downstream'' tasks such as action localisation, which are smaller due to their finer-grained annotations. In this paper, we question this approach, and propose co-finetuning -- simultaneously training a single model on multiple ``upstream'' and ``downstream'' tasks. We demonstrate that co-finetuning outperforms traditional transfer learning when using the same total amount of data, and also show how we can easily extend our approach to multiple ``upstream'' datasets to further improve performance. In particular, co-finetuning significantly improves the performance on rare classes in our downstream task, as it has a regularising effect, and enables the network to learn feature representations that transfer between different datasets. Finally, we observe how co-finetuning with public, video classification datasets, we are able to achieve state-of-the-art results for spatio-temporal action localisation on the challenging AVA and AVA-Kinetics datasets, outperforming recent works which develop intricate models.

cs.CV↗

arXiv2022

Do Trajectories Encode Verb Meaning?

Distributional models learn representations of words from text, but are criticized for their lack of grounding, or the linking of text to the non-linguistic world. Grounded language models have had success in learning to connect concrete categories like nouns and adjectives to the world via images and videos, but can struggle to isolate the meaning of the verbs themselves from the context in which they typically occur. In this paper, we investigate the extent to which trajectories (i.e. the position and rotation of objects over time) naturally encode verb semantics. We build a procedurally generated agent-object-interaction dataset, obtain human annotations for the verbs that occur in this data, and compare several methods for representation learning given the trajectories. We find that trajectories correlate as-is with some verbs (e.g., fall), and that additional abstraction via self-supervised pretraining can further capture nuanced differences in verb meaning (e.g., roll vs. slide).

cs.CL↗

arXiv2022

AVATAR: Unconstrained Audiovisual Speech Recognition

Audio-visual automatic speech recognition (AV-ASR) is an extension of ASR that incorporates visual cues, often from the movements of a speaker's mouth. Unlike works that simply focus on the lip motion, we investigate the contribution of entire visual frames (visual actions, objects, background etc.). This is particularly useful for unconstrained videos, where the speaker is not necessarily visible. To solve this task, we propose a new sequence-to-sequence AudioVisual ASR TrAnsformeR (AVATAR) which is trained end-to-end from spectrograms and full-frame RGB. To prevent the audio stream from dominating training, we propose different word-masking strategies, thereby encouraging our model to pay attention to the visual stream. We demonstrate the contribution of the visual modality on the How2 AV-ASR benchmark, especially in the presence of simulated noise, and show that our model outperforms all other prior work by a large margin. Finally, we also create a new, real-world test bed for AV-ASR called VisSpeech, which demonstrates the contribution of the visual modality under challenging audio conditions.

cs.CV↗

arXiv2022

Spin Accumulation and Longitudinal Spin Diffusion of Magnets

We extend to the longitudinal component of the magnetization the spintronics idea that a magnet near equilibrium can be described by two magnetic variables. One is the usual magnetization $\vec{M}$. The other is the non-equilibrium quantity $\vec{m}$, called the spin accumulation, by which the non-equilibrium spin current can be transported. $\vec{M}$ represents a correlated distribution of a very large number of degrees of freedom, as expressed in some equilibrium distribution function for the excitations; we therefore forbid $\vec{M}$ to diffuse, but we permit $\vec{M}$ to decay. On the other hand, we permit $\vec{m}$, due to spin excitations, to both diffuse and decay. For this physical picture, diffusion from a given region occurs by decay of $\vec{M}$ to $\vec{m}$, then by diffusion of $\vec{m}$, and finally by decay of $\vec{m}$ to $\vec{M}$ in another region. This somewhat slows down the diffusion process. Restricting ourselves to the longitudinal variables $M$ and $m$ with equilibrium properties $M_{eq}=M_{0}+χ_{M\parallel}H$ and $m_{eq}=0$, we argue that the effective energy density must include a new, thermodynamically required exchange constant $λ_{M}=-1/χ_{M\parallel}$. We then develop the macroscopic equations by applying Onsager's irreversible thermodynamics, and use the resulting equations to study the space and time response. At fixed real frequency $ω$ there is, as usual, a single pair of complex wavevectors $\pm k$ but with an unusual dependence on $ω$. At fixed real wavevector, there are two decay constants, as opposed to one in the usual case. Extending the idea that non-equilibrium diffusion in other ordered systems involves a non-equilibrium quantity, this work suggests that in a superconductor the order parameter $Δ$ can decay but not diffuse, but a non-equilibrium gap-like $δ$, due to pair excitations, can both decay and diffuse.

cond-mat.mes-hall↗

arXiv2022

Multiview Transformers for Video Recognition

Video understanding requires reasoning at multiple spatiotemporal resolutions -- from short fine-grained motions to events taking place over longer durations. Although transformer architectures have recently advanced the state-of-the-art, they have not explicitly modelled different spatiotemporal resolutions. To this end, we present Multiview Transformers for Video Recognition (MTV). Our model consists of separate encoders to represent different views of the input video with lateral connections to fuse information across views. We present thorough ablation studies of our model and show that MTV consistently performs better than single-view counterparts in terms of accuracy and computational cost across a range of model sizes. Furthermore, we achieve state-of-the-art results on six standard datasets, and improve even further with large-scale pretraining. Code and checkpoints are available at: https://github.com/google-research/scenic/tree/main/scenic/projects/mtv.

cs.CV↗

arXiv2022

Galactic rotation curves versus ultralight dark matter: A systematic comparison with SPARC data

We look for and place observational constraints on the imprint of ultralight dark matter (ULDM) soliton cores in rotation-dominated galaxies. Extending previous analyses, we find a conservative constraint which disfavors the soliton-host halo relation found in some numerical simulations over a broad range in the ULDM particle mass $m$. Combining the observational constraints with theoretical arguments for the efficiency of soliton formation via gravitational dynamical relaxation, and assuming that the soliton-halo relation is correct, our results disfavor ULDM from comprising 100\% of the total cosmological dark matter in the range $10^{-24}~{\rm eV}\lesssim m\lesssim10^{-20}~{\rm eV}$. The constraints probe the ULDM fraction down to $f\lesssim0.3$ of the total dark matter.

hep-ph↗

arXiv2022

Axion Echos from the Supernova Graveyard

Stimulated decays of axion dark matter, triggered by a source in the sky, could produce a photon flux along the continuation of the line of sight, pointing backward to the source. The strength of this so-called axion "echo" signal depends on the entire history of the source and could still be strong from sources that are dim today but had a large flux density in the past, such as supernova remnants (SNRs). This echo signal turns out to be most observable in the radio band. We study the sensitivity of radio telescopes such as the Square Kilometer Array (SKA) to echo signals generated by SNRs that have already been observed, and show that SKA could reach axion-photon couplings of order $g_{aγγ} \sim \mathcal{O}(10^{-11}) \,\mathrm{GeV}^{-1}$ for axion masses $m_a \lesssim 10^{-5}\;\mathrm{eV}$. In addition, we show projections of the detection reach for signals coming from old SNRs and from newly born supernovae that could be detected in the future. Intriguingly, an observable echo signal could come from old "ghost" SNRs which were very bright in the past but are now so dim that they haven't been observed.

hep-ph↗

arXiv2022

Learning Audio-Video Modalities from Image Captions

A major challenge in text-video and text-audio retrieval is the lack of large-scale training data. This is unlike image-captioning, where datasets are in the order of millions of samples. To close this gap we propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort. Using this pipeline, we create a new large-scale, weakly labelled audio-video captioning dataset consisting of millions of paired clips and captions. We show that training a multimodal transformed based model on this data achieves competitive performance on video retrieval and video captioning, matching or even outperforming HowTo100M pretraining with 20x fewer clips. We also show that our mined clips are suitable for text-audio pretraining, and achieve state of the art results for the task of audio retrieval.

cs.CV↗

arXiv2022

Gravitational Waves from Incomplete Inflationary Phase Transitions

We study the observable implications of an incomplete first order phase transition during inflation. In such a phase transition, the nucleated bubbles do not percolate and instead are continuously produced until the onset of reheating. The process creates an inhomogeneity with a distinct power spectrum that depends on both the physics of the phase transition and the inflationary dynamics. Upon horizon re-entry, this spectrum generates gravitational waves through non-linear effects. This stochastic gravitational wave background is predicted to have unique signatures that may be detectable by future experiments spanning a wide frequency range. The discovery of such a gravitational wave signal would shed a light on the detailed dynamics of inflation.

hep-ph↗

arXiv2022

Constraints on Axions from Cosmic Distance Measurements

Axion couplings to photons could induce photon-axion conversion in the presence of magnetic fields in the Universe. This conversion could impact various cosmic distance measurements, such as luminosity distances to type Ia supernovae and angular distances to galaxy clusters, in different ways. In this paper we consider different combinations of the most up-to-date distance measurements to constrain the axion-photon coupling. Employing the conservative cell magnetic field model for the magnetic fields in the intergalactic medium (IGM) and ignoring the conversion in the intracluster medium (ICM), we find the upper bounds on axion-photon couplings to be around $5 \times 10^{-12}$ (nG/$B$) $\sqrt{\mathrm{Mpc/s}}$ GeV$^{-1}$ for axion masses $m_a$ below $10^{-13}$ eV, where $B$ is the strength of the IGM magnetic field, and $s$ is the comoving size of the magnetic domains. When including the conversion in the ICM, the upper bound is lowered and could reach $5 \times 10^{-13}\, $GeV$^{-1}$ for $m_a < 5 \times 10^{-12}$ eV. While this stronger bound depends on the ICM modeling, it is independent of the strength of the IGM magnetic field, for which there is no direct evidence yet. These constraints could be placed on firmer footing with an enhanced understanding and control of the astrophysical uncertainties associated with the IGM and ICM. All the bounds are determined by the shape of the Hubble rate as a function of redshift reconstructable from various distance measurements, and insensitive to today's Hubble rate, of which there is a tension between early and late cosmological measurements. As an appendix, we discuss the model building challenges of the use of photon-axion conversion to make type Ia supernovae brighter to alleviate the Hubble problem/crisis.

hep-ph↗

arXiv2022

Coherent reaction between molecular and atomic Bose-Einstein condensates: integrable model

We solve a model that describes a stimulated conversion between ultracold bosonic atoms and molecules. The reaction is triggered by a linearly time-dependent transition throughout the Feshbach resonance. Our solution predicts a nonexponential dependence, with a dynamic phase transition, of the reaction efficiency on the transition rate. We find that the emerging phase can have a thermalized energy distribution with the temperature defined by the rate of the transition. This phase, however, has strong purely quantum correlations.

quant-ph↗

arXiv2021

Two-Scalar Bose-Einstein Condensates: From Stars to Galaxies

We study the properties of Bose-Einstein Condensate (BEC) systems consisting of two scalars, focusing on both the case where the BEC is stellar scale as well as the case when it is galactic scale. After studying the stability of such systems and making contact with existing single scalar limits, we undertake a numerical study of the two interacting scalars using Einstein-Klein-Gordon (EKG) equations, including both non-gravitational self-interactions and interactions between the species. We show that the presence of extra scalars and possible interactions between them can leave unique imprints on the BEC system mass profile, especially when the system transitions from being dominated by one scalar to being dominated by the other. At stellar scales (nonlinear regime,) we observe that a repulsive interaction between the two scalars of the type $+ϕ_1^2 ϕ_2^2$ can stabilize the BEC system and support it up to high compactness, a phenomenon only known to exist in the $+ϕ^4$ system. We provide simple analytic understanding of this behavior and point out that it can lead to interesting gravitational wave signals at LIGO-Virgo. At galactic scales, on the other hand, we show that two-scalar BECs can address the scaling problem that arises when one uses ultralight dark matter mass profiles to fit observed galactic core mass profiles. In the end, we construct a particle model of two ultralight scalars with the repulsive $ϕ_1^2 ϕ_2^2$ interaction using collective symmetry breaking. We develop a fast numerical code that utilizes the relaxation method to solve the EKG system, which can be easily generalized to multiple scalars.

astro-ph.CO↗