arXiv subjects

Yilin Wang

Publications and source records attributed to Yilin Wang.

At least 181 records · Page 10Linked to original sources

arXiv2021

High Frame Rate Video Quality Assessment using VMAF and Entropic Differences

The popularity of streaming videos with live, high-action content has led to an increased interest in High Frame Rate (HFR) videos. In this work we address the problem of frame rate dependent Video Quality Assessment (VQA) when the videos to be compared have different frame rate and compression factor. The current VQA models such as VMAF have superior correlation with perceptual judgments when videos to be compared have same frame rates and contain conventional distortions such as compression, scaling etc. However this framework requires additional pre-processing step when videos with different frame rates need to be compared, which can potentially limit its overall performance. Recently, Generalized Entropic Difference (GREED) VQA model was proposed to account for artifacts that arise due to changes in frame rate, and showed superior performance on the LIVE-YT-HFR database which contains frame rate dependent artifacts such as judder, strobing etc. In this paper we propose a simple extension, where the features from VMAF and GREED are fused in order to exploit the advantages of both models. We show through various experiments that the proposed fusion framework results in more efficient features for predicting frame rate dependent video quality. We also evaluate the fused feature set on standard non-HFR VQA databases and obtain superior performance than both GREED and VMAF, indicating the combined feature set captures complimentary perceptual quality information.

cs.MM↗

arXiv2021

ST-GREED: Space-Time Generalized Entropic Differences for Frame Rate Dependent Video Quality Prediction

We consider the problem of conducting frame rate dependent video quality assessment (VQA) on videos of diverse frame rates, including high frame rate (HFR) videos. More generally, we study how perceptual quality is affected by frame rate, and how frame rate and compression combine to affect perceived quality. We devise an objective VQA model called Space-Time GeneRalized Entropic Difference (GREED) which analyzes the statistics of spatial and temporal band-pass video coefficients. A generalized Gaussian distribution (GGD) is used to model band-pass responses, while entropy variations between reference and distorted videos under the GGD model are used to capture video quality variations arising from frame rate changes. The entropic differences are calculated across multiple temporal and spatial subbands, and merged using a learned regressor. We show through extensive experiments that GREED achieves state-of-the-art performance on the LIVE-YT-HFR Database when compared with existing VQA models. The features used in GREED are highly generalizable and obtain competitive performance even on standard, non-HFR VQA databases. The implementation of GREED has been made available online: https://github.com/pavancm/GREED

cs.MM↗

arXiv2021

Flavor Invariants and Renormalization-group Equations in the Leptonic Sector with Massive Majorana Neutrinos

In the present paper, we carry out a systematic study of the flavor invariants and their renormalization-group equations (RGEs) in the leptonic sector with three generations of charged leptons and massive Majorana neutrinos. First, following the approach of the Hilbert series from the invariant theory, we show that there are 34 basic flavor invariants in the generating set, among which 19 invariants are CP-even and the others are CP-odd. Any flavor invariants can be expressed as the polynomials of those 34 basic invariants in the generating set. Second, we explicitly construct all the basic invariants and derive their RGEs, which form a closed system of differential equations as they should. The numerical solutions to the RGEs of the basic flavor invariants have also been found. Furthermore, we demonstrate how to extract physical observables from the basic invariants. Our study is helpful for understanding the algebraic structure of flavor invariants in the leptonic sector, and also provides a novel way to explore leptonic flavor structures.

hep-ph↗

arXiv2021

SSH: A Self-Supervised Framework for Image Harmonization

Image harmonization aims to improve the quality of image compositing by matching the "appearance" (\eg, color tone, brightness and contrast) between foreground and background images. However, collecting large-scale annotated datasets for this task requires complex professional retouching. Instead, we propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. We reformulate the image harmonization problem from a representation fusion perspective, which separately processes the foreground and background examples, to address the background occlusion issue. This framework design allows for a dual data augmentation method, where diverse [foreground, background, pseudo GT] triplets can be generated by cropping an image with perturbations using 3D color lookup tables (LUTs). In addition, we build a real-world harmonization dataset as carefully created by expert users, for evaluation and benchmarking purposes. Our results show that the proposed self-supervised method outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study. Code and dataset are available at \url{https://github.com/VITA-Group/SSHarmonization}.

cs.CV↗

arXiv2021

MUSIQ: Multi-scale Image Quality Transformer

Image quality assessment (IQA) is an important research topic for understanding and improving visual experience. The current state-of-the-art IQA methods are based on convolutional neural networks (CNNs). The performance of CNN-based models is often compromised by the fixed shape constraint in batch training. To accommodate this, the input images are usually resized and cropped to a fixed shape, causing image quality degradation. To address this, we design a multi-scale image quality Transformer (MUSIQ) to process native resolution images with varying sizes and aspect ratios. With a multi-scale image representation, our proposed method can capture image quality at different granularities. Furthermore, a novel hash-based 2D spatial embedding and a scale embedding is proposed to support the positional embedding in the multi-scale representation. Experimental results verify that our method can achieve state-of-the-art performance on multiple large scale IQA datasets such as PaQ-2-PiQ, SPAQ and KonIQ-10k.

cs.CV↗

arXiv2021

Making CNNs Interpretable by Building Dynamic Sequential Decision Forests with Top-down Hierarchy Learning

In this paper, we propose a generic model transfer scheme to make Convlutional Neural Networks (CNNs) interpretable, while maintaining their high classification accuracy. We achieve this by building a differentiable decision forest on top of CNNs, which enjoys two characteristics: 1) During training, the tree hierarchies of the forest are learned in a top-down manner under the guidance from the category semantics embedded in the pre-trained CNN weights; 2) During inference, a single decision tree is dynamically selected from the forest for each input sample, enabling the transferred model to make sequential decisions corresponding to the attributes shared by semantically-similar categories, rather than directly performing flat classification. We name the transferred model deep Dynamic Sequential Decision Forest (dDSDF). Experimental results show that dDSDF not only achieves higher classification accuracy than its conuterpart, i.e., the original CNN, but has much better interpretability, as qualitatively it has plausible hierarchies and quantitatively it leads to more precise saliency maps.

cs.CV↗

arXiv2021

Multimodal Contrastive Training for Visual Representation Learning

We develop an approach to learning visual representations that embraces multimodal data, driven by a combination of intra- and inter-modal similarity preservation objectives. Unlike existing visual pre-training methods, which solve a proxy prediction task in a single domain, our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously, hence improving the quality of learned visual representations. By including multimodal training in a unified framework with different types of contrastive losses, our method can learn more powerful and generic visual features. We first train our model on COCO and evaluate the learned visual representations on various downstream tasks including image classification, object detection, and instance segmentation. For example, the visual representations pre-trained on COCO by our method achieve state-of-the-art top-1 validation accuracy of $55.3\%$ on ImageNet classification, under the common transfer protocol. We also evaluate our method on the large-scale Stock images dataset and show its effectiveness on multi-label image tagging, and cross-modal retrieval tasks.

cs.CV↗

arXiv2021

UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content

Recent years have witnessed an explosion of user-generated content (UGC) videos shared and streamed over the Internet, thanks to the evolution of affordable and reliable consumer capture devices, and the tremendous popularity of social media platforms. Accordingly, there is a great need for accurate video quality assessment (VQA) models for UGC/consumer videos to monitor, control, and optimize this vast content. Blind quality prediction of in-the-wild videos is quite challenging, since the quality degradations of UGC content are unpredictable, complicated, and often commingled. Here we contribute to advancing the UGC-VQA problem by conducting a comprehensive evaluation of leading no-reference/blind VQA (BVQA) features and models on a fixed evaluation architecture, yielding new empirical insights on both subjective video quality studies and VQA model design. By employing a feature selection strategy on top of leading VQA model features, we are able to extract 60 of the 763 statistical features used by the leading models to create a new fusion-based BVQA model, which we dub the \textbf{VID}eo quality \textbf{EVAL}uator (VIDEVAL), that effectively balances the trade-off between VQA performance and efficiency. Our experimental results show that VIDEVAL achieves state-of-the-art performance at considerably lower computational cost than other leading models. Our study protocol also defines a reliable benchmark for the UGC-VQA problem, which we believe will facilitate further research on deep learning-based VQA modeling, as well as perceptually-optimized efficient UGC video processing, transcoding, and streaming. To promote reproducible research and public evaluation, an implementation of VIDEVAL has been made available online: \url{https://github.com/tu184044109/VIDEVAL_release}.

cs.CV↗

arXiv2021

Mask Guided Matting via Progressive Refinement Network

We propose Mask Guided (MG) Matting, a robust matting framework that takes a general coarse mask as guidance. MG Matting leverages a network (PRN) design which encourages the matting model to provide self-guidance to progressively refine the uncertain regions through the decoding process. A series of guidance mask perturbation operations are also introduced in the training to further enhance its robustness to external guidance. We show that PRN can generalize to unseen types of guidance masks such as trimap and low-quality alpha matte, making it suitable for various application pipelines. In addition, we revisit the foreground color prediction problem for matting and propose a surprisingly simple improvement to address the dataset issue. Evaluation on real and synthetic benchmarks shows that MG Matting achieves state-of-the-art performance using various types of guidance inputs. Code and models are available at https://github.com/yucornetto/MGMatting.

cs.CV↗

arXiv2021

Electronic correlations in the normal state of kagome superconductor KV$_3$Sb$_5$

Recently, intensive studies have revealed fascinating physics, such as charge density wave and superconducting states, in the newly synthesized kagome-lattice materials $A$V$_3$Sb$_5$ ($A$=K, Rb, Cs). Despite the rapid progress, fundamental aspects like the magnetic properties and electronic correlations in these materials have not been clearly understood yet. Here, based on the density functional theory plus the single-site dynamical mean-field theory calculations, we investigate the correlated electronic structure and the magnetic properties of the KV$_3$Sb$_5$ family materials in the normal state. We show that these materials are good metals with weak local correlations. The obtained Pauli-like paramagnetism and the absence of local moments are consistent with recent experiment. We reveal that the band crossings around the Fermi level form three groups of nodal lines protected by the spacetime inversion symmetry, each carrying a quantized $π$ Berry phase. Our result suggests that the local correlation strength in these materials appears to be too weak to generate unconventional superconductivity, and non-local electronic correlation might be crucial in this kagome system.

cond-mat.str-el↗

arXiv2021

Classifying Video based on Automatic Content Detection Overview

Video classification and analysis is always a popular and challenging field in computer vision. It is more than just simple image classification due to the correlation with respect to the semantic contents of subsequent frames brings difficulties for video analysis. In this literature review, we summarized some state-of-the-art methods for multi-label video classification. Our goal is first to experimentally research the current widely used architectures, and then to develop a method to deal with the sequential data of frames and perform multi-label classification based on automatic content detection of video.

cs.CV↗

arXiv2021

Regression or Classification? New Methods to Evaluate No-Reference Picture and Video Quality Models

Video and image quality assessment has long been projected as a regression problem, which requires predicting a continuous quality score given an input stimulus. However, recent efforts have shown that accurate quality score regression on real-world user-generated content (UGC) is a very challenging task. To make the problem more tractable, we propose two new methods - binary, and ordinal classification - as alternatives to evaluate and compare no-reference quality models at coarser levels. Moreover, the proposed new tasks convey more practical meaning on perceptually optimized UGC transcoding, or for preprocessing on media processing platforms. We conduct a comprehensive benchmark experiment of popular no-reference quality models on recent in-the-wild picture and video quality datasets, providing reliable baselines for both evaluation methods to support further studies. We hope this work promotes coarse-grained perceptual modeling and its applications to efficient UGC processing.

cs.CV↗

arXiv2020

Electronic Correlations and Absence of Superconductivity in the Collapsed Phase of LaFe$_2$As$_2$

The interplay between structural phase and electronic correlations has been an intriguing topic of research. An prominent example is the pressure-induced uncollapsed to collapsed tetragonal phase transition observed in CaFe$_2$As$_2$, which is accompanied with the emergence of superconductivity in the collapsed phase. Recently, a very similar structural phase transition was discovered in LaFe$_2$As$_2$, but in contrast to CaFe$_2$As$_2$, superconductivity was only observed in the uncollapsed phase, not the collapsed phase. Previous studies have attributed this puzzling observation to the differences in the two materials' band coherence, orbital occupation, and Fermi surface topology. Here, we present a comparative study of LaFe$_2$As$_2$ and CaFe$_2$As$_2$ using the DFT+DMFT method. Surprisingly, we find that although La appears to have a valence higher than Ca, the doped one electron actually primarily resides on the La site. This leads to almost the same total Fe-$3d$ occupancy and electronic correlation strength as well as similar Lifshiftz transition in the Fermi surface topology for the two materials. In addition, we show that the two materials in both structural phases belong to the category of Hund's metals. Our results indicate that the electronic structures of LaFe$_2$As$_2$ and CaFe$_2$As$_2$ are not too different, which further suggest that superconductivity might also be induced in the collapsed phase of LaFe$_2$As$_2$ under similar non-hydrostatic conditions as for CaFe$_2$As$_2$.

cond-mat.str-el↗

arXiv2020

Meticulous Object Segmentation

Compared with common image segmentation tasks targeted at low-resolution images, higher resolution detailed image segmentation receives much less attention. In this paper, we propose and study a task named Meticulous Object Segmentation (MOS), which is focused on segmenting well-defined foreground objects with elaborate shapes in high resolution images (e.g. 2k - 4k). To this end, we propose the MeticulousNet which leverages a dedicated decoder to capture the object boundary details. Specifically, we design a Hierarchical Point-wise Refining (HierPR) block to better delineate object boundaries, and reformulate the decoding process as a recursive coarse to fine refinement of the object mask. To evaluate segmentation quality near object boundaries, we propose the Meticulosity Quality (MQ) score considering both the mask coverage and boundary precision. In addition, we collect a MOS benchmark dataset including 600 high quality images with complex objects. We provide comprehensive empirical evidence showing that MeticulousNet can reveal pixel-accurate segmentation boundaries and is superior to state-of-the-art methods for high resolution object segmentation tasks.

cs.CV↗

arXiv2020

An Overview Of 3D Object Detection

Point cloud 3D object detection has recently received major attention and becomes an active research topic in 3D computer vision community. However, recognizing 3D objects in LiDAR (Light Detection and Ranging) is still a challenge due to the complexity of point clouds. Objects such as pedestrians, cyclists, or traffic cones are usually represented by quite sparse points, which makes the detection quite complex using only point cloud. In this project, we propose a framework that uses both RGB and point cloud data to perform multiclass object recognition. We use existing 2D detection models to localize the region of interest (ROI) on the RGB image, followed by a pixel mapping strategy in the point cloud, and finally, lift the initial 2D bounding box to 3D space. We use the recently released nuScenes dataset---a large-scale dataset contains many data formats---to training and evaluate our proposed architecture.

cs.CV↗

arXiv2020

A Feasible Level Proximal Point Method for Nonconvex Sparse Constrained Optimization

Nonconvex sparse models have received significant attention in high-dimensional machine learning. In this paper, we study a new model consisting of a general convex or nonconvex objectives and a variety of continuous nonconvex sparsity-inducing constraints. For this constrained model, we propose a novel proximal point algorithm that solves a sequence of convex subproblems with gradually relaxed constraint levels. Each subproblem, having a proximal point objective and a convex surrogate constraint, can be efficiently solved based on a fast routine for projection onto the surrogate constraint. We establish the asymptotic convergence of the proposed algorithm to the Karush-Kuhn-Tucker (KKT) solutions. We also establish new convergence complexities to achieve an approximate KKT solution when the objective can be smooth/nonsmooth, deterministic/stochastic and convex/nonconvex with complexity that is on a par with gradient descent for unconstrained optimization problems in respective cases. To the best of our knowledge, this is the first study of the first-order methods with complexity guarantee for nonconvex sparse-constrained problems. We perform numerical experiments to demonstrate the effectiveness of our new model and efficiency of the proposed algorithm for large scale problems.

math.OC↗

arXiv2020

Capturing Video Frame Rate Variations via Entropic Differencing

High frame rate videos are increasingly getting popular in recent years, driven by the strong requirements of the entertainment and streaming industries to provide high quality of experiences to consumers. To achieve the best trade-offs between the bandwidth requirements and video quality in terms of frame rate adaptation, it is imperative to understand the effects of frame rate on video quality. In this direction, we devise a novel statistical entropic differencing method based on a Generalized Gaussian Distribution model expressed in the spatial and temporal band-pass domains, which measures the difference in quality between reference and distorted videos. The proposed design is highly generalizable and can be employed when the reference and distorted sequences have different frame rates. Our proposed model correlates very well with subjective scores in the recently proposed LIVE-YT-HFR database and achieves state of the art performance when compared with existing methodologies.

cs.MM↗

arXiv2020

Kondo signatures in Dirac spin liquids: Non-Abelian bosonization after Chern-Simons fermionization

Quantum impurities serve as in-situ probes of the frustrated quantum magnets, and Dirac spin liquids are an important class of quantum spin liquids. Here, we present a general method, a combination of the Chern-Simons fermionization and the Wess-Zumino-Witten theory, to study the quantum impurity in Dirac spin liquids. Under the Chern-Simons fermionization, the gauge fluctuations are apparently suppressed and the low-energy physics is described by a number of Dirac valleys with valley-dependent pseudospin-momentum locking. The (2+1)D effective theory can be further reduced into the (1+1)D Wess-Zumino-Witten theory by rotational symmetry, where the pseudospin-exchange between Dirac fermions and the impurity can then be solved by the non-Abelian bosonization. Consequently, fixed points of Fermi liquid and non-Fermi liquid are identified, respectively, depending on the relevance of the impurity scattering among the Dirac valleys. This leads to experimental fingerprints for Dirac spin liquids, including a Kondo-induced magneto-thermal effect, a non-monotonous thermal conductivity during the crossover, and an anisotropic spin correlation function.

cond-mat.str-el↗