arXiv subjects

Zhu Li

Publications and source records attributed to Zhu Li.

At least 91 records · Page 5Linked to original sources

arXiv2020

Multiscale Point Cloud Geometry Compression

Recent years have witnessed the growth of point cloud based applications because of its realistic and fine-grained representation of 3D objects and scenes. However, it is a challenging problem to compress sparse, unstructured, and high-precision 3D points for efficient communication. In this paper, leveraging the sparsity nature of point cloud, we propose a multiscale end-to-end learning framework which hierarchically reconstructs the 3D Point Cloud Geometry (PCG) via progressive re-sampling. The framework is developed on top of a sparse convolution based autoencoder for point cloud compression and reconstruction. For the input PCG which has only the binary occupancy attribute, our framework translates it to a downscaled point cloud at the bottleneck layer which possesses both geometry and associated feature attributes. Then, the geometric occupancy is losslessly compressed using an octree codec and the feature attributes are lossy compressed using a learned probabilistic context model.Compared to state-of-the-art Video-based Point Cloud Compression (V-PCC) and Geometry-based PCC (G-PCC) schemes standardized by the Moving Picture Experts Group (MPEG), our method achieves more than 40% and 70% BD-Rate (Bjontegaard Delta Rate) reduction, respectively. Its encoding runtime is comparable to that of G-PCC, which is only 1.5% of V-PCC.

eess.IV↗

arXiv2020

Dense-View GEIs Set: View Space Covering for Gait Recognition based on Dense-View GAN

Gait recognition has proven to be effective for long-distance human recognition. But view variance of gait features would change human appearance greatly and reduce its performance. Most existing gait datasets usually collect data with a dozen different angles, or even more few. Limited view angles would prevent learning better view invariant feature. It can further improve robustness of gait recognition if we collect data with various angles at 1 degree interval. But it is time consuming and labor consuming to collect this kind of dataset. In this paper, we, therefore, introduce a Dense-View GEIs Set (DV-GEIs) to deal with the challenge of limited view angles. This set can cover the whole view space, view angle from 0 degree to 180 degree with 1 degree interval. In addition, Dense-View GAN (DV-GAN) is proposed to synthesize this dense view set. DV-GAN consists of Generator, Discriminator and Monitor, where Monitor is designed to preserve human identification and view information. The proposed method is evaluated on the CASIA-B and OU-ISIR dataset. The experimental results show that DV-GEIs synthesized by DV-GAN is an effective way to learn better view invariant feature. We believe the idea of dense view generated samples will further improve the development of gait recognition.

cs.CV↗

arXiv2020

Referenceless Rate-Distortion Modeling with Learning from Bitstream and Pixel Features

Generally, adaptive bitrates for variable Internet bandwidths can be obtained through multi-pass coding. Referenceless prediction-based methods show practical benefits compared with multi-pass coding to avoid excessive computational resource consumption, especially in low-latency circumstances. However, most of them fail to predict precisely due to the complex inner structure of modern codecs. Therefore, to improve the fidelity of prediction, we propose a referenceless prediction-based R-QP modeling (PmR-QP) method to estimate bitrate by leveraging a deep learning algorithm with only one-pass coding. It refines the global rate-control paradigm in modern codecs on flexibility and applicability with few adjustments as possible. By exploring the potentials of bitstream and pixel features from the prerequisite of one-pass coding, it can reach the expectation of bitrate estimation in terms of precision. To be more specific, we first describe the R-QP relationship curve as a robust quadratic R-QP modeling function derived from the Cauchy-based distribution. Second, we simplify the modeling function by fastening one operational point of the relationship curve received from the coding process. Third, we learn the model parameters from bitstream and pixel features, named them hybrid referenceless features, comprising texture information, hierarchical coding structure, and selected modes in intra-prediction. Extensive experiments demonstrate the proposed method significantly decreases the proportion of samples' bitrate estimation error within 10% by 24.60% on average over the state-of-the-art.

eess.IV↗

arXiv2020

The TianQin project: current progress on science and technology

TianQin is a planned space-based gravitational wave (GW) observatory consisting of three earth orbiting satellites with an orbital radius of about $10^5~{\rm km}$. The satellites will form a equilateral triangle constellation the plane of which is nearly perpendicular to the ecliptic plane. TianQin aims to detect GWs between $10^{-4}~{\rm Hz}$ and $1~{\rm Hz}$ that can be generated by a wide variety of important astrophysical and cosmological sources, including the inspiral of Galactic ultra-compact binaries, the inspiral of stellar-mass black hole binaries, extreme mass ratio inspirals, the merger of massive black hole binaries, and possibly the energetic processes in the very early universe or exotic sources such as cosmic strings. In order to start science operations around 2035, a roadmap called the 0123 plan is being used to bring the key technologies of TianQin to maturity, supported by the construction of a series of research facilities on the ground. Two major projects of the 0123 plan are being carried out. In this process, the team has created a new generation $17~{\rm cm}$ single-body hollow corner-cube retro-reflector which has been launched with the QueQiao satellite on 21 May 2018; a new laser ranging station equipped with a $1.2~{\rm m}$ telescope has been constructed and the station has successfully ranged to all the five retro-reflectors on the Moon; and the TianQin-1 experimental satellite has been launched on 20 December 2019 and the first round result shows that the satellite has exceeded all of its mission requirements.

gr-qc↗

arXiv2020

The First Round Result from the TianQin-1 Satellite

The TianQin-1 satellite (TQ-1), which is the first technology demonstration satellite for the TianQin project, was launched on 20 December 2019. The first round of experiment had been carried out from 21 December 2019 until 1 April 2020. The residual acceleration of the satellite is found to be about $1\times10^{-10}~{\rm m}/{\rm s}^{2}/{\rm Hz}^{1/2}$ at $0.1~{\rm Hz}\,$ and about $5\times10^{-11}~{\rm m}/{\rm s}^{2}/{\rm Hz}^{1/2}$ at $0.05~{\rm Hz}\,$, measured by an inertial sensor with a sensitivity of $5\times10^{-12}~{\rm m}/{\rm s}^{2}/{\rm Hz}^{1/2}$ at $0.1~{\rm Hz}\,$. The micro-Newton thrusters has demonstrated a thrust resolution of $0.1~μ{\rm N}$ and a thrust noise of $0.3~μ{\rm N}/{\rm Hz}^{1/2}$ at $0.1~{\rm Hz}$. The residual noise of the satellite with drag-free control is $3\times10^{-9}~{\rm m}/{\rm s}^{2}/{\rm Hz}^{1/2}$ at $0.1~{\rm Hz}\,$. The noise level of the optical readout system is about $30~{\rm pm}/{\rm Hz}^{1/2}$ at $0.1~{\rm Hz}\,$. The temperature stability at temperature monitoring position is controlled to be about $\pm3~{\rm mK}$ per orbit, and the mismatch between the center-of-mass of the satellite and that of the test mass is measured with a precision of better than $0.1~{\rm mm}$.

physics.ins-det↗

arXiv2020

Hyperspectral Image Classification with Attention Aided CNNs

Convolutional neural networks (CNNs) have been widely used for hyperspectral image classification. As a common process, small cubes are firstly cropped from the hyperspectral image and then fed into CNNs to extract spectral and spatial features. It is well known that different spectral bands and spatial positions in the cubes have different discriminative abilities. If fully explored, this prior information will help improve the learning capacity of CNNs. Along this direction, we propose an attention aided CNN model for spectral-spatial classification of hyperspectral images. Specifically, a spectral attention sub-network and a spatial attention sub-network are proposed for spectral and spatial classification, respectively. Both of them are based on the traditional CNN model, and incorporate attention modules to aid networks focus on more discriminative channels or positions. In the final classification phase, the spectral classification result and the spatial classification result are combined together via an adaptively weighted summation method. To evaluate the effectiveness of the proposed model, we conduct experiments on three standard hyperspectral datasets. The experimental results show that the proposed model can achieve superior performance compared to several state-of-the-art CNN-related models.

eess.IV↗

arXiv2020

Dual Temporal Memory Network for Efficient Video Object Segmentation

Video Object Segmentation (VOS) is typically formulated in a semi-supervised setting. Given the ground-truth segmentation mask on the first frame, the task of VOS is to track and segment the single or multiple objects of interests in the rest frames of the video at the pixel level. One of the fundamental challenges in VOS is how to make the most use of the temporal information to boost the performance. We present an end-to-end network which stores short- and long-term video sequence information preceding the current frame as the temporal memories to address the temporal modeling in VOS. Our network consists of two temporal sub-networks including a short-term memory sub-network and a long-term memory sub-network. The short-term memory sub-network models the fine-grained spatial-temporal interactions between local regions across neighboring frames in video via a graph-based learning framework, which can well preserve the visual consistency of local regions over time. The long-term memory sub-network models the long-range evolution of object via a Simplified-Gated Recurrent Unit (S-GRU), making the segmentation be robust against occlusions and drift errors. In our experiments, we show that our proposed method achieves a favorable and competitive performance on three frequently-used VOS datasets, including DAVIS 2016, DAVIS 2017 and Youtube-VOS in terms of both speed and accuracy.

cs.CV↗

arXiv2020

Classification of Hyperspectral and LiDAR Data Using Coupled CNNs

In this paper, we propose an efficient and effective framework to fuse hyperspectral and Light Detection And Ranging (LiDAR) data using two coupled convolutional neural networks (CNNs). One CNN is designed to learn spectral-spatial features from hyperspectral data, and the other one is used to capture the elevation information from LiDAR data. Both of them consist of three convolutional layers, and the last two convolutional layers are coupled together via a parameter sharing strategy. In the fusion phase, feature-level and decision-level fusion methods are simultaneously used to integrate these heterogeneous features sufficiently. For the feature-level fusion, three different fusion strategies are evaluated, including the concatenation strategy, the maximization strategy, and the summation strategy. For the decision-level fusion, a weighted summation strategy is adopted, where the weights are determined by the classification accuracy of each output. The proposed model is evaluated on an urban data set acquired over Houston, USA, and a rural one captured over Trento, Italy. On the Houston data, our model can achieve a new record overall accuracy of 96.03%. On the Trento data, it achieves an overall accuracy of 99.12%. These results sufficiently certify the effectiveness of our proposed model.

cs.CV↗

arXiv2019

Kernel Dependence Regularizers and Gaussian Processes with Applications to Algorithmic Fairness

Current adoption of machine learning in industrial, societal and economical activities has raised concerns about the fairness, equity and ethics of automated decisions. Predictive models are often developed using biased datasets and thus retain or even exacerbate biases in their decisions and recommendations. Removing the sensitive covariates, such as gender or race, is insufficient to remedy this issue since the biases may be retained due to other related covariates. We present a regularization approach to this problem that trades off predictive accuracy of the learned models (with respect to biased labels) for the fairness in terms of statistical parity, i.e. independence of the decisions from the sensitive covariates. In particular, we consider a general framework of regularized empirical risk minimization over reproducing kernel Hilbert spaces and impose an additional regularizer of dependence between predictors and sensitive covariates using kernel-based measures of dependence, namely the Hilbert-Schmidt Independence Criterion (HSIC) and its normalized version. This approach leads to a closed-form solution in the case of squared loss, i.e. ridge regression. Moreover, we show that the dependence regularizer has an interpretation as modifying the corresponding Gaussian process (GP) prior. As a consequence, a GP model with a prior that encourages fairness to sensitive variables can be derived, allowing principled hyperparameter selection and studying of the relative relevance of covariates under fairness constraints. Experimental results in synthetic examples and in real problems of income and crime prediction illustrate the potential of the approach to improve fairness of automated decisions.

stat.ML↗

arXiv2019

Video-based compression for plenoptic point clouds

The plenoptic point cloud that has multiple colors from various directions, is a more complete representation than the general point cloud that usually has only one color. It is more realistic but also brings a larger volume of data that needs to be compressed efficiently. The state-of-the-art method to compress the plenoptic point cloud is an extension of the region-based adaptive hierarchical transform (RAHT). As far as we can see, in addition to RAHT, the video-based point cloud compression (V-PCC) is also an efficient point cloud compression method. However, to the best of our knowledge, no works have used a video-based solution to compress the plenoptic point cloud yet. In this paper, we first extend the V-PCC to support the plenoptic point cloud compression by generating multiple attribute videos. Then based on the observation that these videos from multiple views have very high correlations, we propose encoding them using multiview high efficiency video coding. We further propose a block-based padding method that unifies the unoccupied attribute pixels from different views to reduce their bit cost. The proposed algorithms are implemented in the V-PCC reference software. The experimental results show that the proposed algorithms can bring significant bitrate savings compared with the state-of-the-art method for plenoptic point cloud compression.

eess.IV↗

arXiv2019

Deep AutoEncoder-based Lossy Geometry Compression for Point Clouds

Point cloud is a fundamental 3D representation which is widely used in real world applications such as autonomous driving. As a newly-developed media format which is characterized by complexity and irregularity, point cloud creates a need for compression algorithms which are more flexible than existing codecs. Recently, autoencoders(AEs) have shown their effectiveness in many visual analysis tasks as well as image compression, which inspires us to employ it in point cloud compression. In this paper, we propose a general autoencoder-based architecture for lossy geometry point cloud compression. To the best of our knowledge, it is the first autoencoder-based geometry compression codec that directly takes point clouds as input rather than voxel grids or collections of images. Compared with handcrafted codecs, this approach adapts much more quickly to previously unseen media contents and media formats, meanwhile achieving competitive performance. Our architecture consists of a pointnet-based encoder, a uniform quantizer, an entropy estimation block and a nonlinear synthesis transformation module. In lossy geometry compression of point cloud, results show that the proposed method outperforms the test model for categories 1 and 3 (TMC13) published by MPEG-3DG group on the 125th meeting, and on average a 73.15\% BD-rate gain is achieved.

cs.CV↗

arXiv2019

Occupancy-map-based rate distortion optimization for video-based point cloud compression

The state-of-the-art video-based point cloud compression scheme projects the 3D point cloud to 2D patch by patch and organizes the patches into frames to compress them using the efficient video compression scheme. Such a scheme shows a good trade-off between the number of points projected and the video continuity to utilize the video compression scheme. However, some unoccupied pixels between different patches are compressed using almost the same quality with the occupied pixels, which will lead to the waste of lots of bits since the unoccupied pixels are useless for the reconstructed point cloud. In this paper, we propose to consider only the rate instead of the rate distortion cost for the unoccupied pixels during the rate distortion optimization process. The proposed scheme can be applied to both the geometry and attribute frames. The experimental results show that the proposed algorithm can achieve an average of 11.9% and 15.4% bitrate savings for the geometry and attribute, respectively.

cs.MM↗

arXiv2018

Hybrid Point Cloud Attribute Compression Using Slice-based Layered Structure and Block-based Intra Prediction

Point cloud compression is a key enabler for the emerging applications of immersive visual communication, autonomous driving and smart cities, etc. In this paper, we propose a hybrid point cloud attribute compression scheme built on an original layered data structure. First, a slice-partition scheme and geometry-adaptive k dimensional-tree (kd-tree) method are devised to generate the four-layer structure. Second, we introduce an efficient block-based intra prediction scheme containing a DC prediction mode and several angular modes, in order to exploit the spatial correlation between adjacent points. Third, an adaptive transform scheme based on Graph Fourier Transform (GFT) is Lagrangian optimized to achieve better transform efficiency. The Lagrange multiplier is off-line derived based on the statistics of color attribute coding. Last but not least, multiple reordering scan modes are dedicated to improve coding efficiency for entropy coding. In intra-frame compression of point cloud color attributes, results demonstrate that our method performs better than the state-of-the-art region-adaptive hierarchical transform (RAHT) system, and on average a 29.37$\%$ BD-rate gain is achieved. Comparing with the test model for category 1 (TMC1) anchor's coding results, which were recently published by MPEG-3DG group on 121st meeting, a 16.37$\%$ BD-rate gain is obtained.

cs.MM↗

arXiv2017

Attribute Compression of 3D Point Clouds Using Laplacian Sparsity Optimized Graph Transform

3D sensing and content capture have made significant progress in recent years and the MPEG standardization organization is launching a new project on immersive media with point cloud compression (PCC) as one key corner stone. In this work, we introduce a new binary tree based point cloud content partition and explore the graph signal processing tools, especially the graph transform with optimized Laplacian sparsity, to achieve better energy compaction and compression efficiency. The resulting rate-distortion operating points are convex-hull optimized over the existing Lagrangian solutions. Simulation results with the latest high quality point cloud content captured from the MPEG PCC demonstrated the transform efficiency and rate-distortion (R-D) optimal potential of the proposed solutions.

cs.MM↗

arXiv2017

Robust Emotion Recognition from Low Quality and Low Bit Rate Video: A Deep Learning Approach

Emotion recognition from facial expressions is tremendously useful, especially when coupled with smart devices and wireless multimedia applications. However, the inadequate network bandwidth often limits the spatial resolution of the transmitted video, which will heavily degrade the recognition reliability. We develop a novel framework to achieve robust emotion recognition from low bit rate video. While video frames are downsampled at the encoder side, the decoder is embedded with a deep network model for joint super-resolution (SR) and recognition. Notably, we propose a novel max-mix training strategy, leading to a single "One-for-All" model that is remarkably robust to a vast range of downsampling factors. That makes our framework well adapted for the varied bandwidths in real transmission scenarios, without hampering scalability or efficiency. The proposed framework is evaluated on the AVEC 2016 benchmark, and demonstrates significantly improved stand-alone recognition performance, as well as rate-distortion (R-D) performance, than either directly recognizing from LR frames, or separating SR and recognition.

cs.CV↗

arXiv2017

Co-projection-plane based 3-D padding for polyhedron projection for 360-degree video

The polyhedron projection for 360-degree video is becoming more and more popular since it can lead to much less geometry distortion compared with the equirectangular projection. However, in the polyhedron projection, we can observe very obvious texture discontinuity in the area near the face boundary. Such a texture discontinuity may lead to serious quality degradation when motion compensation crosses the discontinuous face boundary. To solve this problem, in this paper, we first propose to fill the corresponding neighboring faces in the suitable positions as the extension of the current face to keep approximated texture continuity. Then a co-projection-plane based 3-D padding method is proposed to project the reference pixels in the neighboring face to the current face to guarantee exact texture continuity. Under the proposed scheme, the reference pixel is always projected to the same plane with the current pixel when performing motion compensation so that the texture discontinuity problem can be solved. The proposed scheme is implemented in the reference software of High Efficiency Video Coding. Compared with the existing method, the proposed algorithm can significantly improve the rate-distortion performance. The experimental results obviously demonstrate that the texture discontinuity in the face boundary can be well handled by the proposed algorithm.

cs.MM↗

arXiv2017

Projection based advanced motion model for cubic mapping for 360-degree video

This paper proposes a novel advanced motion model to handle the irregular motion for the cubic map projection of 360-degree video. Since the irregular motion is mainly caused by the projection from the sphere to the cube map, we first try to project the pixels in both the current picture and reference picture from unfolding cube back to the sphere. Then through utilizing the characteristic that most of the motions in the sphere are uniform, we can derive the relationship between the motion vectors of various pixels in the unfold cube. The proposed advanced motion model is implemented in the High Efficiency Video Coding reference software. Experimental results demonstrate that quite obvious performance improvement can be achieved for the sequences with obvious motions.

cs.MM↗

arXiv2016

Pseudo Sequence based 2-D hierarchical reference structure for Light-Field Image Compression

In this paper, we present a novel pseudo sequence based 2-D hierarchical reference structure for light-field image compression. In the proposed scheme, we first decompose the light-field image into multiple views and organize them into a 2-D coding structure according to the spatial coordinates of the corresponding microlens. Then we mainly develop three technologies to optimize the 2-D coding structure. First, we divide all the views into four quadrants, and all the views are encoded one quadrant after another to reduce the reference buffer size as much as possible. Inside each quadrant, all the views are encoded hierarchically to fully exploit the correlations between different views. Second, we propose to use the distance between the current view and its reference views as the criteria for selecting better reference frames for each inter view. Third, we propose to use the spatial relative positions between different views to achieve more accurate motion vector scaling. The whole scheme is implemented in the reference software of High Efficiency Video Coding. The experimental results demonstrate that the proposed novel pseudo-sequence based 2-D hierarchical structure can achieve maximum 14.2% bit-rate savings compared with the state-of-the-art light-field image compression method.

cs.MM↗