arXiv subjects

Chen Sun

Publications and source records attributed to Chen Sun.

At least 181 records · Page 10Linked to original sources

arXiv2021

DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets

Due to the stochasticity of human behaviors, predicting the future trajectories of road agents is challenging for autonomous driving. Recently, goal-based multi-trajectory prediction methods are proved to be effective, where they first score over-sampled goal candidates and then select a final set from them. However, these methods usually involve goal predictions based on sparse pre-defined anchors and heuristic goal selection algorithms. In this work, we propose an anchor-free and end-to-end trajectory prediction model, named DenseTNT, that directly outputs a set of trajectories from dense goal candidates. In addition, we introduce an offline optimization-based technique to provide multi-future pseudo-labels for our final online model. Experiments show that DenseTNT achieves state-of-the-art performance, ranking 1st on the Argoverse motion forecasting benchmark and being the 1st place winner of the 2021 Waymo Open Dataset Motion Prediction Challenge.

cs.CV↗

arXiv2021

Masking Modalities for Cross-modal Video Retrieval

Pre-training on large scale unlabelled datasets has shown impressive performance improvements in the fields of computer vision and natural language processing. Given the advent of large-scale instructional video datasets, a common strategy for pre-training video encoders is to use the accompanying speech as weak supervision. However, as speech is used to supervise the pre-training, it is never seen by the video encoder, which does not learn to process that modality. We address this drawback of current pre-training methods, which fail to exploit the rich cues in spoken language. Our proposal is to pre-train a video encoder using all the available video modalities as supervision, namely, appearance, sound, and transcribed speech. We mask an entire modality in the input and predict it using the other two modalities. This encourages each modality to collaborate with the others, and our video encoder learns to process appearance and audio as well as speech. We show the superior performance of our "modality masking" pre-training approach for video retrieval on the How2R, YouCook2 and Condensed Movies datasets.

cs.CV↗

arXiv2021

ViViT: A Video Vision Transformer

We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatio-temporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks. To facilitate further research, we release code at https://github.com/google-research/scenic/tree/main/scenic/projects/vivit

cs.CV↗

arXiv2021

Does Vision-and-Language Pretraining Improve Lexical Grounding?

Linguistic representations derived from text alone have been criticized for their lack of grounding, i.e., connecting words to their meanings in the physical world. Vision-and-Language (VL) models, trained jointly on text and image or video data, have been offered as a response to such criticisms. However, while VL pretraining has shown success on multimodal tasks such as visual question answering, it is not yet known how the internal linguistic representations themselves compare to their text-only counterparts. This paper compares the semantic representations learned via VL vs. text-only pretraining for two recent VL models using a suite of analyses (clustering, probing, and performance on a commonsense question answering task) in a language-only setting. We find that the multimodal models fail to significantly outperform the text-only variants, suggesting that future work is required if multimodal pretraining is to be pursued as a means of improving NLP in general.

cs.CL↗

arXiv2021

Learning Temporal Dynamics from Cycles in Narrated Video

Learning to model how the world changes as time elapses has proven a challenging problem for the computer vision community. We propose a self-supervised solution to this problem using temporal cycle consistency jointly in vision and language, training on narrated video. Our model learns modality-agnostic functions to predict forward and backward in time, which must undo each other when composed. This constraint leads to the discovery of high-level transitions between moments in time, since such transitions are easily inverted and shared across modalities. We justify the design of our model with an ablation study on different configurations of the cycle consistency problem. We then show qualitatively and quantitatively that our approach yields a meaningful, high-level model of the future and past. We apply the learned dynamics model without further training to various tasks, such as predicting future action and temporally ordering sets of images. Project page: https://dave.ml/mmcc

cs.CV↗

arXiv2021

Learning Video Representations from Textual Web Supervision

Videos on the Internet are paired with pieces of text, such as titles and descriptions. This text typically describes the most important content in the video, such as the objects in the scene and the actions being performed. Based on this observation, we propose to use text as a method for learning video representations. To accomplish this, we propose a data collection process and use it to collect 70M video clips shared publicly on the Internet, and we then train a model to pair each video with its associated text. We evaluate the model on several down-stream action recognition tasks, including Kinetics, HMDB-51, and UCF-101. We find that this approach is an effective method of pre-training video representations. Specifically, it outperforms all existing methods for self-supervised and cross-modal video representation learning.

cs.CV↗

arXiv2021

Episodic Transformer for Vision-and-Language Navigation

Interaction and navigation defined by natural language instructions in dynamic environments pose significant challenges for neural agents. This paper focuses on addressing two challenges: handling long sequence of subtasks, and understanding complex human instructions. We propose Episodic Transformer (E.T.), a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions. To improve training, we leverage synthetic instructions as an intermediate representation that decouples understanding the visual appearance of an environment from the variations of natural language instructions. We demonstrate that encoding the history with a transformer is critical to solve compositional tasks, and that pretraining and joint training with synthetic instructions further improve the performance. Our approach sets a new state of the art on the challenging ALFRED benchmark, achieving 38.4% and 8.5% task success rates on seen and unseen test splits.

cs.CV↗

arXiv2021

Composable Augmentation Encoding for Video Representation Learning

We focus on contrastive methods for self-supervised video representation learning. A common paradigm in contrastive learning is to construct positive pairs by sampling different data views for the same instance, with different data instances as negatives. These methods implicitly assume a set of representational invariances to the view selection mechanism (eg, sampling frames with temporal shifts), which may lead to poor performance on downstream tasks which violate these invariances (fine-grained video action recognition that would benefit from temporal information). To overcome this limitation, we propose an 'augmentation aware' contrastive learning framework, where we explicitly provide a sequence of augmentation parameterisations (such as the values of the time shifts used to create data views) as composable augmentation encodings (CATE) to our model when projecting the video representations for contrastive learning. We show that representations learned by our method encode valuable information about specified spatial or temporal augmentation, and in doing so also achieve state-of-the-art performance on a number of video benchmarks.

cs.CV↗

arXiv2021

Discrete-Valued Neural Communication

Deep learning has advanced from fully connected architectures to structured models organized into components, e.g., the transformer composed of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes. In structured models, an interesting question is how to conduct dynamic and possibly sparse communication among the separate components. Here, we explore the hypothesis that restricting the transmitted information among components to discrete representations is a beneficial bottleneck. The motivating intuition is human language in which communication occurs through discrete symbols. Even though individuals have different understandings of what a "cat" is based on their specific experiences, the shared discrete token makes it possible for communication among individuals to be unimpeded by individual differences in internal representation. To discretize the values of concepts dynamically communicated among specialist components, we extend the quantization mechanism from the Vector-Quantized Variational Autoencoder to multi-headed discretization with shared codebooks and use it for discrete-valued neural communication (DVNC). Our experiments show that DVNC substantially improves systematic generalization in a variety of architectures -- transformers, modular architectures, and graph neural networks. We also show that the DVNC is robust to the choice of hyperparameters, making the method very useful in practice. Moreover, we establish a theoretical justification of our discretization process, proving that it has the ability to increase noise robustness and reduce the underlying dimensionality of the model.

cs.LG↗

arXiv2021

HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps

High Definition (HD) maps are maps with precise definitions of road lanes with rich semantics of the traffic rules. They are critical for several key stages in an autonomous driving system, including motion forecasting and planning. However, there are only a small amount of real-world road topologies and geometries, which significantly limits our ability to test out the self-driving stack to generalize onto new unseen scenarios. To address this issue, we introduce a new challenging task to generate HD maps. In this work, we explore several autoregressive models using different data representations, including sequence, plain graph, and hierarchical graph. We propose HDMapGen, a hierarchical graph generation model capable of producing high-quality and diverse HD maps through a coarse-to-fine approach. Experiments on the Argoverse dataset and an in-house dataset show that HDMapGen significantly outperforms baseline methods. Additionally, we demonstrate that HDMapGen achieves high scalability and efficiency.

cs.CV↗

arXiv2021

Challenges for an axion explanation of the muon $g-2$ measurement

The discrepancy between the muon $g-2$ measurement and the Standard Model prediction points to new physics around or below the weak scale. It is tantalizing to consider the loop effects of a heavy axion (in the general sense, also known as an axion-like particle) coupling to leptons and photons as an explanation for this discrepancy. We provide an updated analysis of the necessary couplings, including two-loop contributions, and find that the new physics operators point to an axion decay constant on the order of 10s of GeV. This poses major problems for such an explanation, as the axion couplings to leptons and photons must be generated at low scales. We outline some possibilities for how such couplings can arise, and find that these scenarios predict new charged matter at or below the weak scale and new scalars can mix with the Higgs boson, raising numerous phenomenological challenges. These scenarios also all predict additional contributions to the muon $g-2$ itself, calling the initial application of the axion effective theory into question. We conclude that there is little reason to favor an axion explanation of the muon $g-2$ measurement relative to other models postulating new weak-scale matter.

hep-ph↗

arXiv2021

Local Metrics for Multi-Object Tracking

This paper introduces temporally local metrics for Multi-Object Tracking. These metrics are obtained by restricting existing metrics based on track matching to a finite temporal horizon, and provide new insight into the ability of trackers to maintain identity over time. Moreover, the horizon parameter offers a novel, meaningful mechanism by which to define the relative importance of detection and association, a common dilemma in applications where imperfect association is tolerable. It is shown that the historical Average Tracking Accuracy (ATA) metric exhibits superior sensitivity to association, enabling its proposed local variant, ALTA, to capture a wide range of characteristics. In particular, ALTA is better equipped to identify advances in association independent of detection. The paper further presents an error decomposition for ATA that reveals the impact of four distinct error types and is equally applicable to ALTA. The diagnostic capabilities of ALTA are demonstrated on the MOT 2017 and Waymo Open Dataset benchmarks.

cs.CV↗

arXiv2021

Unified Graph Structured Models for Video Understanding

Accurate video understanding involves reasoning about the relationships between actors, objects and their environment, often over long temporal intervals. In this paper, we propose a message passing graph neural network that explicitly models these spatio-temporal relations and can use explicit representations of objects, when supervision is available, and implicit representations otherwise. Our formulation generalises previous structured models for video understanding, and allows us to study how different design choices in graph structure and representation affect the model's performance. We demonstrate our method on two different tasks requiring relational reasoning in videos -- spatio-temporal action detection on AVA and UCF101-24, and video scene graph classification on the recent Action Genome dataset -- and achieve state-of-the-art results on all three datasets. Furthermore, we show quantitatively and qualitatively how our method is able to more effectively model relationships between relevant entities in the scene.

cs.CV↗

arXiv2021

Axions: From Magnetars and Neutron Star Mergers to Beam Dumps and BECs

We review topics in searches for axion-like-particles (ALPs), covering material that is complementary to other recent reviews. The first half of our review covers ALPs in the extreme environments of neutron star cores, the magnetospheres of highly magnetized neutron stars (magnetars), and in neutron star mergers. The focus is on possible signals of ALPs in the photon spectrum of neutron stars and gravitational wave/electromagnetic signals from neutron star mergers. We then review recent developments in laboratory-produced ALP searches, focusing mainly on accelerator-based facilities including beam-dump type experiments and collider experiments. We provide a general-purpose discussion of the ALP search pipeline from production to detection, in steps, and our discussion is straightforwardly applicable to most beam-dump type and reactor experiments. We end with a selective look at the rapidly developing field of ultralight dark matter, specifically the formation of Bose-Einstein Condensates (BECs). We review the properties of BECs of ultralight dark matter and bridge these properties with developments in numerical simulations, and ultimately with their impact on experimental searches.

hep-ph↗

arXiv2021

From Machine Learning to Transfer Learning in Laser-Induced Breakdown Spectroscopy: the Case of Rock Analysis for Mars Exploration

With the ChemCam instrument, laser-induced breakdown spectroscopy (LIBS) has successively contributed to Mars exploration by determining elemental compositions of the soil, crust and rocks. Two new lunched missions, Chinese Tianwen 1 and American Perseverance, will further increase the number of LIBS instruments on Mars after the planned landings in spring 2021. Such unprecedented situation requires a reinforced research effort on the methods of LIBS spectral data treatment. Although the matrix effects correspond to a general issue in LIBS, they become accentuated in the case of rock analysis for Mars exploration, because of the large variation of rock composition leading to the chemical matrix effect, and the difference in morphology between laboratory standard samples (in pressed pellet, glass or ceramics) used to establish calibration models and natural rocks encountered on Mars, leading to the physical matric effect. The chemical matrix effect has been tackled in the ChemCam project with large sets of laboratory standard samples offering a good representation of various compositions of Mars rocks. The present work deals with the physical matrix effect which is still expecting a satisfactory solution. The approach consists in introducing transfer learning in LIBS data treatment. For the specific case of total alkali-silica (TAS) classification of natural rocks, the results show a significant improvement of the prediction capacity of pellet sample-based models when trained together with suitable information from rocks in a procedure of transfer learning. The correct classification rate of rocks increases from 33.3% with a machine learning model to 83.3% with a transfer learning model.

physics.app-ph↗

arXiv2020

What Makes for Good Views for Contrastive Learning?

Contrastive learning between multiple views of the data has recently achieved state of the art performance in the field of self-supervised representation learning. Despite its success, the influence of different view choices has been less studied. In this paper, we use theoretical and empirical analysis to better understand the importance of view selection, and argue that we should reduce the mutual information (MI) between views while keeping task-relevant information intact. To verify this hypothesis, we devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI. We also consider data augmentation as a way to reduce MI, and show that increasing data augmentation indeed leads to decreasing MI and improves downstream classification accuracy. As a by-product, we achieve a new state-of-the-art accuracy on unsupervised pre-training for ImageNet classification ($73\%$ top-1 linear readout with a ResNet-50). In addition, transferring our models to PASCAL VOC object detection and COCO instance segmentation consistently outperforms supervised pre-training. Code:http://github.com/HobbitLong/PyContrast

cs.CV↗

arXiv2020

Bottom Contact Metal Oxide Interface Modification Improving the Efficiency of Organic Light Emitting Diodes

The performance of solution-processed organic light emitting diodes (OLEDs) is often limited by non-uniform contacts. In this work, we introduce Ni-containing solution-processed metal oxide (MO) interfacial layers inserted between indium tin oxide (ITO) and poly(3,4-ethylenedioxythiophene):poly(styrene sulfonate) (PEDOT:PSS) to improve the bottom electrode contact for OLEDs using the poly(p-phenylene vinylene) (PPV) derivative Super-Yellow (SY) as an emission layer. For ITO/Ni-containing MO/PEDOT:PSS bottom electrode structures we show enhanced wetting properties that result in an improved OLED device efficiency. Best performance is achieved using a Cu-Li co-doped spinel nickel cobaltite [(Cu-Li):NiCo2O4], for which the current efficiency and luminous efficacy of SY OLEDs increased, respectively, by 12% and 11% from the values obtained for standard devices without a Ni-containing MO interface modification between ITO and PEDOT:PSS. The enhanced performance was attributed to the improved morphology of PEDOT:PSS, which consequently increased the hole injection capability of the optimized ITO/(Cu-Li):NiCo2O4/PEDOT:PSS electrode.

cond-mat.mtrl-sci↗

arXiv2020

PH-Pfaffian order in a translationally and rotationally invariant system

The PH-Pfaffian topological order has been proposed as a candidate order for the $ν=5/2$ quantum Hall effect. The PH-Pfaffian liquid is known to be the ground state in several coupled wire and coupled stripe constructions. No translationally and rotationally invariant models with the PH-Pfaffian ground state have been identified so far. By employing anyon condensation on top of a topological order, allowed in an isotropic system, we argue that the PH-Pfaffian order is possible in the presence of rotational and translational symmetries.

cond-mat.mes-hall↗