arXiv subjects

Zhanyu Wang

Publications and source records attributed to Zhanyu Wang.

25 records · Page 2Linked to original sources

arXiv2021

CLIP4Caption ++: Multi-CLIP for Video Caption

This report describes our solution to the VALUE Challenge 2021 in the captioning task. Our solution, named CLIP4Caption++, is built on X-Linear/X-Transformer, which is an advanced model with encoder-decoder architecture. We make the following improvements on the proposed CLIP4Caption++: We employ an advanced encoder-decoder model architecture X-Transformer as our main framework and make the following improvements: 1) we utilize three strong pre-trained CLIP models to extract the text-related appearance visual features. 2) we adopt the TSN sampling strategy for data enhancement. 3) we involve the video subtitle information to provide richer semantic information. 3) we introduce the subtitle information, which fuses with the visual features as guidance. 4) we design word-level and sentence-level ensemble strategies. Our proposed method achieves 86.5, 148.4, 64.5 CIDEr scores on VATEX, YC2C, and TVC datasets, respectively, which shows the superior performance of our proposed CLIP4Caption++ on all three datasets.

cs.CV↗

arXiv2021

CLIP4Caption: CLIP for Video Caption

Video captioning is a challenging task since it requires generating sentences describing various diverse and complex videos. Existing video captioning models lack adequate visual representation due to the neglect of the existence of gaps between videos and texts. To bridge this gap, in this paper, we propose a CLIP4Caption framework that improves video captioning based on a CLIP-enhanced video-text matching network (VTM). This framework is taking full advantage of the information from both vision and language and enforcing the model to learn strongly text-correlated video features for text generation. Besides, unlike most existing models using LSTM or GRU as the sentence decoder, we adopt a Transformer structured decoder network to effectively learn the long-range visual and language dependency. Additionally, we introduce a novel ensemble strategy for captioning tasks. Experimental results demonstrate the effectiveness of our method on two datasets: 1) on MSR-VTT dataset, our method achieved a new state-of-the-art result with a significant gain of up to 10% in CIDEr; 2) on the private test data, our method ranking 2nd place in the ACM MM multimedia grand challenge 2021: Pre-training for Video Understanding Challenge. It is noted that our model is only trained on the MSR-VTT dataset.

cs.CV↗

arXiv2021

Variance Reduction on General Adaptive Stochastic Mirror Descent

In this work, we investigate the idea of variance reduction by studying its properties with general adaptive mirror descent algorithms in nonsmooth nonconvex finite-sum optimization problems. We propose a simple yet generalized framework for variance reduced adaptive mirror descent algorithms named SVRAMD and provide its convergence analysis in both the nonsmooth nonconvex problem and the P-L conditioned problem. We prove that variance reduction reduces the SFO complexity of adaptive mirror descent algorithms and thus accelerates their convergence. In particular, our general theory implies that variance reduction can be applied to algorithms using time-varying step sizes and self-adaptive algorithms such as AdaGrad and RMSProp. Moreover, the convergence rates of SVRAMD recover the best existing rates of non-adaptive variance reduced mirror descent algorithms without complicated algorithmic components. Extensive experiments in deep learning validate our theoretical findings.

stat.ML↗

arXiv2020

Directional Pruning of Deep Neural Networks

In the light of the fact that the stochastic gradient descent (SGD) often finds a flat minimum valley in the training loss, we propose a novel directional pruning method which searches for a sparse minimizer in or close to that flat region. The proposed pruning method does not require retraining or the expert knowledge on the sparsity level. To overcome the computational formidability of estimating the flat directions, we propose to use a carefully tuned $\ell_1$ proximal gradient algorithm which can provably achieve the directional pruning with a small learning rate after sufficient training. The empirical results demonstrate the promising results of our solution in highly sparse regime (92% sparsity) among many existing pruning methods on the ResNet50 with the ImageNet, while using only a slightly higher wall time and memory footprint than the SGD. Using the VGG16 and the wide ResNet 28x10 on the CIFAR-10 and CIFAR-100, we demonstrate that our solution reaches the same minima valley as the SGD, and the minima found by our solution and the SGD do not deviate in directions that impact the training loss. The code that reproduces the results of this paper is available at https://github.com/donlan2710/gRDA-Optimizer/tree/master/directional_pruning.

cs.LG↗

arXiv2020

The Sample Complexity of Meta Sparse Regression

This paper addresses the meta-learning problem in sparse linear regression with infinite tasks. We assume that the learner can access several similar tasks. The goal of the learner is to transfer knowledge from the prior tasks to a similar but novel task. For p parameters, size of the support set k , and l samples per task, we show that T \in O (( k log(p) ) /l ) tasks are sufficient in order to recover the common support of all tasks. With the recovered support, we can greatly reduce the sample complexity for estimating the parameter of the novel task, i.e., l \in O (1) with respect to T and p . We also prove that our rates are minimax optimal. A key difference between meta-learning and the classical multi-task learning, is that meta-learning focuses only on the recovery of the parameters of the novel task, while multi-task learning estimates the parameter of all tasks, which requires l to grow with T . Instead, our efficient meta-learning estimator allows for l to be constant with respect to T (i.e., few-shot learning).

cs.LG↗

arXiv2016

BAUM: A DNA Assembler by Adaptive Unique Mapping and Local Overlap-Layout-Consensus

Genome assembly from the high-throughput sequencing (HTS) reads is a fundamental yet challenging computational problem. An intrinsic challenge is the uncertainty caused by the widespread repetitive elements. Here we get around the uncertainty using the notion of uniquely mapped (UM) reads, which motivated the design of a new assembler BAUM. It mainly consists of two types of iterations. The first type of iterations constructs initial contigs from a reference, say a genome of a species that could be quite distant, by adaptive read mapping, filtration by the reference's unique regions, and reference updating. A statistical test is proposed to split the layouts at possible structural variation sites. The second type of iterations includes mapping, scaffolding/contig-extension, and contig merging. We extend each contig by locally assembling the reads whose mates are uniquely mapped to an end of the contig. Instead of the de Bruijn graph method, we take the overlap-layout-consensus (OLC) paradigm. The OLC is implemented by parallel computation, and has linear complexity with respect to the number of contigs. The adjacent extended contigs are merged if their alignment is confirmed by the adjusted gap distance. Throughout the assembling, the mapping criterion is selected by probabilistic calculations. These innovations can be used complementary to the existing de novo assemblers. Applying this novel method to the assembly of wild rice Oryza longistaminata genome, we achieved much improved contig N50, 18.8k, compared with other assemblers. The assembly was further validated by contigs constructed from an independent library of long 454 reads.

q-bio.GN↗

arXiv2015

Electronic and optical properties of novel carbon allotropes

The phonon properties, electronic structures and optical properties of novel carbon allotropes, such as monolayer penta-graphene (PG), double-layer PG and T12-carbon, were explored by means of first-principles calculations. Results of phonon calculations demonstrate that these exotic carbon allotropes are dynamically stable. In addition, the bulk T12 phase is an indirect-gap semiconductor having a bandgap of ~4.89 eV. Whereas the bulk material transforms to a 2D phase, the monolayer and double-layer PG become quasi-direct gap semiconductors with smaller band gaps of ~2.64 eV and ~3.27eV, respectively. Furthermore, the partial charge density analysis indicates that the 2D phases retain part of the electronic characteristics of the T12 phase. The linear photon energy-dependent dielectric functions and related optical properties including refractive index, extinction coefficient, absorption spectrum, reflectivity, and energy loss spectrum were also computed and discussed. The structural estimation obtained as well as other findings are in agreement with existing theoretical data. The calculated results are beneficial to the practical applications of these exotic carbon allotropes in optoelectronics and electronics.

cond-mat.mtrl-sci↗