arXiv subjects

Tao Mei

Publications and source records attributed to Tao Mei.

At least 217 records · Page 12Linked to original sources

arXiv2016

Deep Quantization: Encoding Convolutional Activations with Deep Generative Model

Deep convolutional neural networks (CNNs) have proven highly effective for visual recognition, where learning a universal representation from activations of convolutional layer plays a fundamental problem. In this paper, we present Fisher Vector encoding with Variational Auto-Encoder (FV-VAE), a novel deep architecture that quantizes the local activations of convolutional layer in a deep generative model, by training them in an end-to-end manner. To incorporate FV encoding strategy into deep generative models, we introduce Variational Auto-Encoder model, which steers a variational inference and learning in a neural network which can be straightforwardly optimized using standard stochastic gradient method. Different from the FV characterized by conventional generative models (e.g., Gaussian Mixture Model) which parsimoniously fit a discrete mixture model to data distribution, the proposed FV-VAE is more flexible to represent the natural property of data for better generalization. Extensive experiments are conducted on three public datasets, i.e., UCF101, ActivityNet, and CUB-200-2011 in the context of video action recognition and fine-grained image classification, respectively. Superior results are reported when compared to state-of-the-art representations. Most remarkably, our proposed FV-VAE achieves to-date the best published accuracy of 94.2% on UCF101.

cs.CV↗

arXiv2016

Video Captioning with Transferred Semantic Attributes

Automatically generating natural language descriptions of videos plays a fundamental challenge for computer vision community. Most recent progress in this problem has been achieved through employing 2-D and/or 3-D Convolutional Neural Networks (CNN) to encode video content and Recurrent Neural Networks (RNN) to decode a sentence. In this paper, we present Long Short-Term Memory with Transferred Semantic Attributes (LSTM-TSA)---a novel deep architecture that incorporates the transferred semantic attributes learnt from images and videos into the CNN plus RNN framework, by training them in an end-to-end manner. The design of LSTM-TSA is highly inspired by the facts that 1) semantic attributes play a significant contribution to captioning, and 2) images and videos carry complementary semantics and thus can reinforce each other for captioning. To boost video captioning, we propose a novel transfer unit to model the mutually correlated attributes learnt from images and videos. Extensive experiments are conducted on three public datasets, i.e., MSVD, M-VAD and MPII-MD. Our proposed LSTM-TSA achieves to-date the best published performance in sentence generation on MSVD: 52.8% and 74.0% in terms of BLEU@4 and CIDEr-D. Superior results when compared to state-of-the-art methods are also reported on M-VAD and MPII-MD.

cs.CV↗

arXiv2016

Boosting Image Captioning with Attributes

Automatically describing an image with a natural language has been an emerging challenge in both fields of computer vision and natural language processing. In this paper, we present Long Short-Term Memory with Attributes (LSTM-A) - a novel architecture that integrates attributes into the successful Convolutional Neural Networks (CNNs) plus Recurrent Neural Networks (RNNs) image captioning framework, by training them in an end-to-end manner. To incorporate attributes, we construct variants of architectures by feeding image representations and attributes into RNNs in different ways to explore the mutual but also fuzzy relationship between them. Extensive experiments are conducted on COCO image captioning dataset and our framework achieves superior results when compared to state-of-the-art deep models. Most remarkably, we obtain METEOR/CIDEr-D of 25.2%/98.6% on testing data of widely used and publicly available splits in (Karpathy & Fei-Fei, 2015) when extracting image representations by GoogleNet and achieve to date top-1 performance on COCO captioning Leaderboard.

cs.CV↗

arXiv2016

Free Hilbert Transforms

We study analogues of classical Hilbert transforms as fourier multipliers on free groups. We prove their complete boundedness on non commutative $L^p$ spaces associated with the free group von Neumann algebras for all $1<p<\infty$. This implies that the decomposition of the free group $\F_\infty$ into reduced words starting with distinct free generators is completely unconditional in $L^p$. We study the case of Voiculescu's amalgamated free products of von Neumann algebras as well. As by-products, we obtain a positive answer to a compactness-problem posed by Ozawa, a length independent estimate for Junge-Parcet-Xu's free Rosenthal inequality, a Littlewood-Paley-Stein type inequality for geodesic paths of free groups, and a length reduction formula for $L^p$-norms of free group von Neumann algebras.

math.OA↗

arXiv2016

Storytelling of Photo Stream with Bidirectional Multi-thread Recurrent Neural Network

Visual storytelling aims to generate human-level narrative language (i.e., a natural paragraph with multiple sentences) from a photo streams. A typical photo story consists of a global timeline with multi-thread local storylines, where each storyline occurs in one different scene. Such complex structure leads to large content gaps at scene transitions between consecutive photos. Most existing image/video captioning methods can only achieve limited performance, because the units in traditional recurrent neural networks (RNN) tend to "forget" the previous state when the visual sequence is inconsistent. In this paper, we propose a novel visual storytelling approach with Bidirectional Multi-thread Recurrent Neural Network (BMRNN). First, based on the mined local storylines, a skip gated recurrent unit (sGRU) with delay control is proposed to maintain longer range visual information. Second, by using sGRU as basic units, the BMRNN is trained to align the local storylines into the global sequential timeline. Third, a new training scheme with a storyline-constrained objective function is proposed by jointly considering both global and local matches. Experiments on three standard storytelling datasets show that the BMRNN model outperforms the state-of-the-art methods.

cs.CV↗

arXiv2016

A diffusion and clustering-based approach for finding coherent motions and understanding crowd scenes

This paper addresses the problem of detecting coherent motions in crowd scenes and presents its two applications in crowd scene understanding: semantic region detection and recurrent activity mining. It processes input motion fields (e.g., optical flow fields) and produces a coherent motion filed, named as thermal energy field. The thermal energy field is able to capture both motion correlation among particles and the motion trends of individual particles which are helpful to discover coherency among them. We further introduce a two-step clustering process to construct stable semantic regions from the extracted time-varying coherent motions. These semantic regions can be used to recognize pre-defined activities in crowd scenes. Finally, we introduce a cluster-and-merge process which automatically discovers recurrent activities in crowd scenes by clustering and merging the extracted coherent motions. Experiments on various videos demonstrate the effectiveness of our approach.

cs.CV↗

arXiv2015

Large-scale Online Feature Selection for Ultra-high Dimensional Sparse Data

Feature selection with large-scale high-dimensional data is important yet very challenging in machine learning and data mining. Online feature selection is a promising new paradigm that is more efficient and scalable than batch feature section methods, but the existing online approaches usually fall short in their inferior efficacy as compared with batch approaches. In this paper, we present a novel second-order online feature selection scheme that is simple yet effective, very fast and extremely scalable to deal with large-scale ultra-high dimensional sparse data streams. The basic idea is to improve the existing first-order online feature selection methods by exploiting second-order information for choosing the subset of important features with high confidence weights. However, unlike many second-order learning methods that often suffer from extra high computational cost, we devise a novel smart algorithm for second-order online feature selection using a MaxHeap-based approach, which is not only more effective than the existing first-order approaches, but also significantly more efficient and scalable for large-scale feature selection with ultra-high dimensional sparse data, as validated from our extensive experiments. Impressively, on a billion-scale synthetic dataset (1-billion dimensions, 1-billion nonzero features, and 1-million samples), our new algorithm took only 8 minutes on a single PC, which is orders of magnitudes faster than traditional batch approaches. \url{http://arxiv.org/abs/1409.7794}

cs.LG↗

arXiv2015

User-Curated Image Collections: Modeling and Recommendation

Most state-of-the-art image retrieval and recommendation systems predominantly focus on individual images. In contrast, socially curated image collections, condensing distinctive yet coherent images into one set, are largely overlooked by the research communities. In this paper, we aim to design a novel recommendation system that can provide users with image collections relevant to individual personal preferences and interests. To this end, two key issues need to be addressed, i.e., image collection modeling and similarity measurement. For image collection modeling, we consider each image collection as a whole in a group sparse reconstruction framework and extract concise collection descriptors given the pretrained dictionaries. We then consider image collection recommendation as a dynamic similarity measurement problem in response to user's clicked image set, and employ a metric learner to measure the similarity between the image collection and the clicked image set. As there is no previous work directly comparable to this study, we implement several competitive baselines and related methods for comparison. The evaluations on a large scale Pinterest data set have validated the effectiveness of our proposed methods for modeling and recommending image collections.

cs.MM↗

arXiv2015

Tree-based Visualization and Optimization for Image Collection

The visualization of an image collection is the process of displaying a collection of images on a screen under some specific layout requirements. This paper focuses on an important problem that is not well addressed by the previous methods: visualizing image collections into arbitrary layout shapes while arranging images according to user-defined semantic or visual correlations (e.g., color or object category). To this end, we first propose a property-based tree construction scheme to organize images of a collection into a tree structure according to user-defined properties. In this way, images can be adaptively placed with the desired semantic or visual correlations in the final visualization layout. Then, we design a two-step visualization optimization scheme to further optimize image layouts. As a result, multiple layout effects including layout shape and image overlap ratio can be effectively controlled to guarantee a satisfactory visualization. Finally, we also propose a tree-transfer scheme such that visualization layouts can be adaptively changed when users select different "images of interest". We demonstrate the effectiveness of our proposed approach through the comparisons with state-of-the-art visualization techniques.

cs.MM↗

arXiv2015

Jointly Modeling Embedding and Translation to Bridge Video and Language

Automatically describing video content with natural language is a fundamental challenge of multimedia. Recurrent Neural Networks (RNN), which models sequence dynamics, has attracted increasing attention on visual interpretation. However, most existing approaches generate a word locally with given previous words and the visual content, while the relationship between sentence semantics and visual content is not holistically exploited. As a result, the generated sentences may be contextually correct but the semantics (e.g., subjects, verbs or objects) are not true. This paper presents a novel unified framework, named Long Short-Term Memory with visual-semantic Embedding (LSTM-E), which can simultaneously explore the learning of LSTM and visual-semantic embedding. The former aims to locally maximize the probability of generating the next word given previous words and visual content, while the latter is to create a visual-semantic embedding space for enforcing the relationship between the semantics of the entire sentence and visual content. Our proposed LSTM-E consists of three components: a 2-D and/or 3-D deep convolutional neural networks for learning powerful video representation, a deep RNN for generating sentences, and a joint embedding model for exploring the relationships between visual content and sentence semantics. The experiments on YouTube2Text dataset show that our proposed LSTM-E achieves to-date the best reported performance in generating natural sentences: 45.3% and 31.0% in terms of BLEU@4 and METEOR, respectively. We also demonstrate that LSTM-E is superior in predicting Subject-Verb-Object (SVO) triplets to several state-of-the-art techniques.

cs.CV↗

arXiv2015

Complete boundedness of the Heat Semigroups on the von Neumann Algebra of hyperbolic groups

We prove that $(λ_g\mapsto e^{-t|g|^r}λ_g)_{t>0}$ defines a completely bounded semigroup of multipliers on the von Neuman algebra of hyperbolic groups for all real number $r$. One ingredient in the proof is the observation that a construction of Ozawa allows to characterize the radial multipliers that are bounded on every hyperbolic graph, partially generalizing results of Haagerup--Steenstrup--Szwarc and Wysoczański. Another ingredient is an upper estimate of trace class norms for Hankel matrices, which is based on Peller's characterization of such norms.

math.OA↗

arXiv2014

Smooth Fourier multipliers on group von Neumann algebras

We investigate Fourier multipliers on the compact dual of arbitrary discrete groups. Our main result is a Hörmander-Mihlin multiplier theorem for finite-dimensional cocycles with optimal smoothness condition. We also find Littlewood-Paley type inequalities in group von Neumann algebras, prove $L_p$ estimates for noncommutative Riesz transforms and characterize $L_\infty \to \mathrm{BMO}$ boundedness for radial Fourier multipliers. The key novelties of our approach are to exploit group cocycles and cross products in Fourier multiplier theory in conjunction with BMO spaces associated to semigroups of operators and a noncommutative generalization of Calderón-Zygmund theory.

math.CA↗

arXiv2014

Noncommutative Riesz transforms -- Dimension free bounds and Fourier multipliers

We obtain dimension free estimates for noncommutative Riesz transforms associated to conditionally negative length functions on group von Neumann algebras. This includes Poisson semigroups, beyond Bakry's results in the commutative setting. Our proof is inspired by Pisier's method and a new Khintchine inequality for crossed products. New estimates include Riesz transforms associated to fractional laplacians in $\mathbb{R}^n$ (where Meyer's conjecture fails) or to the word length of free groups. Lust-Piquard's work for discrete laplacians on LCA groups is also generalized in several ways. In the context of Fourier multipliers, we will prove that Hörmander-Mihlin multipliers are Littlewood-Paley averages of our Riesz transforms. This is highly surprising in the Euclidean and (most notably) noncommutative settings. As application we provide new Sobolev/Besov type smoothness conditions. The Sobolev type condition we give refines the classical one and yields dimension free constants. Our results hold for arbitrary unimodular groups.

math.CA↗

arXiv2014

Large BMO spaces vs interpolation

In this paper we introduce a class of BMO spaces which interpolate with $L_p$ and are sufficiently large to serve as endpoints for new singular integral operators. More precisely, let $(Ω, Σ, μ)$ be a $σ$-finite measure space. Consider two filtrations of $Σ$ by successive refinement of two atomic $σ$-algebras $Σ_\mathrm{a}, Σ_\mathrm{b}$ having trivial intersection. Construct the corresponding truncated martingale BMO spaces. Then, the intersection seminorm only leaves out constants and we provide a quite flexible condition on $(Σ_\mathrm{a}, Σ_\mathrm{b})$ so that the resulting space interpolates with $L_p$ in the expected way. In the presence of a metric $d$, we obtain endpoint estimates for Calderón-Zygmund operators on $(Ω,μ, d)$ under additional conditions on $(Σ_\mathrm{a}, Σ_\mathrm{b})$. These are weak forms of the \lq isoperimetric\rq${}$ and the \lq locally doubling\rq${}$ properties of Carbonaro/Mauceri/Meda which admit less concentration at the boundary. Examples of particular interest include densities of the form $e^{\pm |x|^α}$ for any $α> 0$ or $(1 + |x|^β)^{-1}$ for any $β\gtrsim n^{3/2}$. A (limited) comparison with Tolsa's RBMO is also possible. On the other hand, a more intrinsic formulation yields a Calderón-Zygmund theory adapted to regular filtrations over $(Σ_\mathrm{a}, Σ_\mathrm{b})$ without using a metric. This generalizes well-known estimates for perfect dyadic and Haar shift operators. In contrast to previous approaches, ours extends to matrix-valued functions (via recent results from noncommutative martingale theory) for which only limited results are known and no satisfactory nondoubling theory exists so far.

math.CA↗

arXiv2013

Image Tag Refinement by Regularized Latent Dirichlet Allocation

Tagging is nowadays the most prevalent and practical way to make images searchable. However, in reality many manually-assigned tags are irrelevant to image content and hence are not reliable for applications. A lot of recent efforts have been conducted to refine image tags. In this paper, we propose to do tag refinement from the angle of topic modeling and present a novel graphical model, regularized Latent Dirichlet Allocation (rLDA). In the proposed approach, tag similarity and tag relevance are jointly estimated in an iterative manner, so that they can benefit from each other, and the multi-wise relationships among tags are explored. Moreover, both the statistics of tags and visual affinities of images in the corpus are explored to help topic modeling. We also analyze the superiority of our approach from the deep structure perspective. The experiments on tag ranking and image retrieval demonstrate the advantages of the proposed method.

cs.IR↗

arXiv2013

An invitation to harmonic analysis associated with semigroups of operators

This article is an introduction to our recent work in harmonic analysis associated with semigroups of operators, in the effort of finding a noncommutative Calderón-Zygmund theory for von Neumann algebras. The classical CZ theory has been traditionally developed on metric measure spaces satisfying additional regularity properties. In the lack of such metrics -or with very little information on the metric- Markov semigroups of operators appear to be the right substitutes of classical metric/geometric tools in harmonic analysis. Our approach is particularly useful in the noncommutative setting but it is also valid in classical/commutative frameworks.

math.CA↗

arXiv2012

John-Nirenberg inequality and atomic decomposition for noncommutative martingales

In this paper, we study the John-Nirenberg inequality for BMO and the atomic decomposition for H1 of noncommutative martingales. We first establish a crude version of the column (resp. row) John-Nirenberg inequality for all 0 < p < \infty. By an extreme point property of Lp -space for 0 < p \leq 1, we then obtain a fine version of this in equality. The latter corresponds exactly to the classical John-Nirenberg inequality and enables us to obtain an exponential integrability inequality like in the classical case. These results extend and improve Junge and Musat's John-Nirenberg inequality. By duality, we obtain the corresponding q-atomic decomposition for different Hardy spaces H1 for all 1<q\leq\infty, which extends the 2-atomic decomposition previously obtained by Bekjan et al. Finally, we give a negative answer to a question posed by Junge and Musat about BMO.

math.FA↗

arXiv2012

An H1-BMO duality theory for semigroups of operators

Let (M,μ) be a sigma-finite measure space. Let (T_t) be a semigroup of positive preserving maps on (M,μ) with standard assumptions. We prove a H_1-BMO duality theory with assumptions only on T_t. The BMO is defined as spaces of functions f such that the L_\infty norm of sup_tT_t|f-T_tf|^2 is finite. The H1 is defined by square functions of P. A. Meyer's gradient form. Our argument does not rely on any geometric/metric structure of M nor on the kernel of the semigroups of operators. This abstract argument allows to extend our main results to the noncommutative setting, e.g. the case where L_\infty(M,μ) is replaced by von Neuman algebras with a semifinite trace. We also prove a Carleson embedding theorem for semigroups of operators.

math.CA↗