arXiv subjects

Zhan Ma

Publications and source records attributed to Zhan Ma.

104 records · Page 6Linked to original sources

arXiv2020

Extreme Image Coding via Multiscale Autoencoders With Generative Adversarial Optimization

We propose a MultiScale AutoEncoder(MSAE) based extreme image compression framework to offer visually pleasing reconstruction at a very low bitrate. Our method leverages the "priors" at different resolution scale to improve the compression efficiency, and also employs the generative adversarial network(GAN) with multiscale discriminators to perform the end-to-end trainable rate-distortion optimization. We compare the perceptual quality of our reconstructions with traditional compression algorithms using High-Efficiency Video Coding(HEVC) based Intra Profile and JPEG2000 on the public Cityscapes and ADE20K datasets, demonstrating the significant subjective quality improvement.

eess.IV↗

arXiv2019

Learned Video Compression via Joint Spatial-Temporal Correlation Exploration

Traditional video compression technologies have been developed over decades in pursuit of higher coding efficiency. Efficient temporal information representation plays a key role in video coding. Thus, in this paper, we propose to exploit the temporal correlation using both first-order optical flow and second-order flow prediction. We suggest an one-stage learning approach to encapsulate flow as quantized features from consecutive frames which is then entropy coded with adaptive contexts conditioned on joint spatial-temporal priors to exploit second-order correlations. Joint priors are embedded in autoregressive spatial neighbors, co-located hyper elements and temporal neighbors using ConvLSTM recurrently. We evaluate our approach for the low-delay scenario with High-Efficiency Video Coding (H.265/HEVC), H.264/AVC and another learned video compression method, following the common test settings. Our work offers the state-of-the-art performance, with consistent gains across all popular test sequences.

eess.IV↗

arXiv2019

Learned Point Cloud Geometry Compression

This paper presents a novel end-to-end Learned Point Cloud Geometry Compression (a.k.a., Learned-PCGC) framework, to efficiently compress the point cloud geometry (PCG) using deep neural networks (DNN) based variational autoencoders (VAE). In our approach, PCG is first voxelized, scaled and partitioned into non-overlapped 3D cubes, which is then fed into stacked 3D convolutions for compact latent feature and hyperprior generation. Hyperpriors are used to improve the conditional probability modeling of latent features. A weighted binary cross-entropy (WBCE) loss is applied in training while an adaptive thresholding is used in inference to remove unnecessary voxels and reduce the distortion. Objectively, our method exceeds the geometry-based point cloud compression (G-PCC) algorithm standardized by well-known Moving Picture Experts Group (MPEG) with a significant performance margin, e.g., at least 60% BD-Rate (Bjontegaard Delta Rate) gains, using common test datasets. Subjectively, our method has presented better visual quality with smoother surface reconstruction and appealing details, in comparison to all existing MPEG standard compliant PCC methods. Our method requires about 2.5MB parameters in total, which is a fairly small size for practical implementation, even on embedded platform. Additional ablation studies analyze a variety of aspects (e.g., cube size, kernels, etc) to explore the application potentials of our learned-PCGC.

cs.CV↗

arXiv2019

Compressive Sampling for Array Cameras

While design of high performance lenses and image sensors has long been the focus of camera development, the size, weight and power of image data processing components is currently the primary barrier to radical improvements in camera resolution. Here we show that Deep-Learning- Aided Compressive Sampling (DLACS) can reduce operating power on camera-head electronics by 20x. Traditional compressive sampling has to date been primarily applied in the physical sensor layer, we show here that with aid from deep learning algorithms, compressive sampling offers unique power management advantages in digital layer compression.

eess.IV↗

arXiv2019

Learned Quality Enhancement via Multi-Frame Priors for HEVC Compliant Low-Delay Applications

Networked video applications, e.g., video conferencing, often suffer from poor visual quality due to unexpected network fluctuation and limited bandwidth. In this paper, we have developed a Quality Enhancement Network (QENet) to reduce the video compression artifacts, leveraging the spatial and temporal priors generated by respective multi-scale convolutions spatially and warped temporal predictions in a recurrent fashion temporally. We have integrated this QENet as a standard-alone post-processing subsystem to the High Efficiency Video Coding (HEVC) compliant decoder. Experimental results show that our QENet demonstrates the state-of-the-art performance against default in-loop filters in HEVC and other deep learning based methods with noticeable objective gains in Peak-Signal-to-Noise Ratio (PSNR) and subjective gains visually.

eess.IV↗

arXiv2019

Non-local Attention Optimized Deep Image Compression

This paper proposes a novel Non-Local Attention Optimized Deep Image Compression (NLAIC) framework, which is built on top of the popular variational auto-encoder (VAE) structure. Our NLAIC framework embeds non-local operations in the encoders and decoders for both image and latent feature probability information (known as hyperprior) to capture both local and global correlations, and apply attention mechanism to generate masks that are used to weigh the features for the image and hyperprior, which implicitly adapt bit allocation for different features based on their importance. Furthermore, both hyperpriors and spatial-channel neighbors of the latent features are used to improve entropy coding. The proposed model outperforms the existing methods on Kodak dataset, including learned (e.g., Balle2019, Balle2018) and conventional (e.g., BPG, JPEG2000, JPEG) image compression methods, for both PSNR and MS-SSIM distortion metrics.

eess.IV↗

arXiv2019

Gated Context Model with Embedded Priors for Deep Image Compression

A deep image compression scheme is proposed in this paper, offering the state-of-the-art compression efficiency, against the traditional JPEG, JPEG2000, BPG and those popular learning based methodologies. This is achieved by a novel conditional probably model with embedded priors which can accurately approximate the entropy rate for rate-distortion optimization. It utilizes three separable stacks to eliminate the blind spots in the receptive field for better probability prediction and computation reduction. Those embedded priors can be further used to help the image reconstruction when fused with latent features, after passing through the proposed information compensation network (ICN). Residual learning with generalized divisive normalization (GDN) based activation is used in our encoder and decoder with fast convergence rate and efficient performance. We have evaluated our model and other methods using rate-distortion criteria, where distortion is measured by multi-scale structural similarity (MS-SSIM). We have also discussed the impacts of various distortion metrics on the reconstructed image quality. Besides, a field study on perceptual quality is also given via a dedicated subjective assessment, to compare the efficiency of our proposed methods and other conventional image compression methods.

eess.IV↗

arXiv2019

Neural Video Compression using Spatio-Temporal Priors

The pursuit of higher compression efficiency continuously drives the advances of video coding technologies. Fundamentally, we wish to find better "predictions" or "priors" that are reconstructed previously to remove the signal dependency efficiently and to accurately model the signal distribution for entropy coding. In this work, we propose a neural video compression framework, leveraging the spatial and temporal priors, independently and jointly to exploit the correlations in intra texture, optical flow based temporal motion and residuals. Spatial priors are generated using downscaled low-resolution features, while temporal priors (from previous reference frames and residuals) are captured using a convolutional neural network based long-short term memory (ConvLSTM) structure in a temporal recurrent fashion. All of these parts are connected and trained jointly towards the optimal rate-distortion performance. Compared with the High-Efficiency Video Coding (HEVC) Main Profile (MP), our method has demonstrated averaged 38% Bjontegaard-Delta Rate (BD-Rate) improvement using standard common test sequences, where the distortion is multi-scale structural similarity (MS-SSIM).

eess.IV↗

arXiv2018

Deep Image Compression via End-to-End Learning

We present a lossy image compression method based on deep convolutional neural networks (CNNs), which outperforms the existing BPG, WebP, JPEG2000 and JPEG as measured via multi-scale structural similarity (MS-SSIM), at the same bit rate. Currently, most of the CNNs based approaches train the network using a L2 loss between the reconstructions and the ground-truths in the pixel domain, which leads to over-smoothing results and visual quality degradation especially at a very low bit rate. Therefore, we improve the subjective quality with the combination of a perception loss and an adversarial loss additionally. To achieve better rate-distortion optimization (RDO), we also introduce an easy-to-hard transfer learning when adding quantization error and rate constraint. Finally, we evaluate our method on public Kodak and the Test Dataset P/M released by the Computer Vision Lab of ETH Zurich, resulting in averaged 7.81% and 19.1% BD-rate reduction over BPG, respectively.

eess.IV↗

arXiv2018

Perceptual Quality Assessment of Immersive Images Considering Peripheral Vision Impact

Conventional images/videos are often rendered within the central vision area of the human visual system (HVS) with uniform quality. Recent virtual reality (VR) device with head mounted display (HMD) extends the field of view (FoV) significantly to include both central and peripheral vision areas. It exhibits the unequal image quality sensation among these areas because of the non-uniform distribution of photoreceptors on our retina. We propose to study the sensation impact on the image subjective quality with respect to the eccentric angle $θ$ across different vision areas. Often times, image quality is controlled by the quantization stepsize $q$ and spatial resolution $s$, separately and jointly. Therefore, the sensation impact can be understood by exploring the $q$ and/or $s$ in terms of the $θ$, resulting in self-adaptive analytical models that have shown quite impressive accuracy through independent cross validations. These models can further be applied to give different quality weights at different regions, so as to significantly reduce the transmission data size but without subjective quality loss. As demonstrated in a gigapixel imaging system, we have shown that the image rendering can be speed up about 10$\times$ with the model guided unequal quality scales, in comparison to the the legacy scheme with uniform quality scales everywhere.

cs.MM↗

arXiv2018

Multispectral Focal Stack Acquisition Using A Chromatic Aberration Enlarged Camera

Capturing more information, e.g. geometry and material, using optical cameras can greatly help the perception and understanding of complex scenes. This paper proposes a novel method to capture the spectral and light field information simultaneously. By using a delicately designed chromatic aberration enlarged camera, the spectral-varying slices at different depths of the scene can be easily captured. Afterwards, the multispectral focal stack, which is composed of a stack of multispectral slice images focusing on different depths, can be recovered from the spectral-varying slices by using a Local Linear Transformation (LLT) based algorithm. The experiments verify the effectiveness of the proposed method.

eess.IV↗

arXiv2018

Viewport Adaptation-Based Immersive Video Streaming: Perceptual Modeling and Applications

Immersive video offers the freedom to navigate inside virtualized environment. Instead of streaming the bulky immersive videos entirely, a viewport (also referred to as field of view, FoV) adaptive streaming is preferred. We often stream the high-quality content within current viewport, while reducing the quality of representation elsewhere to save the network bandwidth consumption. Consider that we could refine the quality when focusing on a new FoV, in this paper, we model the perceptual impact of the quality variations (through adapting the quantization stepsize and spatial resolution) with respect to the refinement duration, and yield a product of two closed-form exponential functions that well explain the joint quantization and resolution induced quality impact. Analytical model is cross-validated using another set of data, where both Pearson and Spearman's rank correlation coefficients are close to 0.98. Our work is devised to optimize the adaptive FoV streaming of the immersive video under limited network resource. Numerical results show that our proposed model significantly improves the quality of experience of users, with about 9.36\% BD-Rate (Bjontegaard Delta Rate) improvement on average as compared to other representative methods, particularly under the limited bandwidth.

cs.MM↗

arXiv2017

Driving dynamic colloidal assembly using eccentric self-propelled colloids

Designing protocols to dynamically direct the self-assembly of colloidal particles has become an important direction in soft matter physics because of the promising applications in fabrication of dynamic responsive functional materials. Here using computer simulations, we found that in the mixture of passive colloids and eccentric self-propelled active particles, when the eccentricity and self-propulsion of active particles are high enough, the eccentric active particles can push passive colloids to form a large dense dynamic cluster, and the system undergoes a novel dynamic demixing transition. Our simulations show that the dynamic demixing occurs when the eccentric active particles move much faster than the passive particles such that the dynamic trajectories of different active particles can overlap with each other while passive particles are depleted from the dynamic trajectories of active particles. Our results suggest that this is in analogy to the entropy driven demixing in colloid-polymer mixtures, in which polymer random coils can overlap with each other while deplete the colloids. More interestingly, we find that by fixing the passive colloid composition at certain value, with increasing the density, the system undergoes an intriguing re-entrant mixing, and the demixing only occurs within certain intermediate density range. This suggests a new way of designing active matter to drive the self-assembly of passive colloids and fabricate dynamic responsive materials.

cond-mat.soft↗

arXiv2012

Rate Model for Compressed Video Considering Impacts Of Spatial, Temporal and Amplitude Resolutions and Its Applications for Video Coding and Adaptation

In this paper, we investigate the impacts of spatial, temporal and amplitude resolution (STAR) on the bit rate of a compressed video. We propose an analytical rate model in terms of the quantization stepsize, frame size and frame rate. Experimental results reveal that the increase of the video rate as the individual resolution increases follows a power function. Hence, the proposed model expresses the rate as the product of power functions of the quantization stepsize, frame size and frame rate, respectively. The proposed rate model is analytically tractable, requiring only four content dependent parameters. We also propose methods for predicting the model parameters from content features that can be computed from original video. Simulation results show that model predicted rates fit the measured data very well with high Pearson correlation (PC) and small relative root mean square error (RRMSE). The same model function works for different coding scenarios (including scalable and non-scalable video, temporal prediction using either hierarchical B or IPPP structure, etc.) with very high accuracy (average PC $>$ 0.99), but the values of model parameters differ. Using the proposed rate model and the quality model introduced in a separate work, we show how to optimize the STAR for a given rate constraint, which is important for both encoder rate control and scalable video adaptation. Furthermore, we demonstrate how to order the spatial, temporal and amplitude layers of a scalable video in a rate-quality optimized way.

cs.MM↗