arXiv subjects

Jie Liang

Publications and source records attributed to Jie Liang.

At least 73 records · Page 4Linked to original sources

arXiv2024

Learned Image Compression with Dual-Branch Encoder and Conditional Information Coding

Recent advancements in deep learning-based image compression are notable. However, prevalent schemes that employ a serial context-adaptive entropy model to enhance rate-distortion (R-D) performance are markedly slow. Furthermore, the complexities of the encoding and decoding networks are substantially high, rendering them unsuitable for some practical applications. In this paper, we propose two techniques to balance the trade-off between complexity and performance. First, we introduce two branching coding networks to independently learn a low-resolution latent representation and a high-resolution latent representation of the input image, discriminatively representing the global and local information therein. Second, we utilize the high-resolution latent representation as conditional information for the low-resolution latent representation, furnishing it with global information, thus aiding in the reduction of redundancy between low-resolution information. We do not utilize any serial entropy models. Instead, we employ a parallel channel-wise auto-regressive entropy model for encoding and decoding low-resolution and high-resolution latent representations. Experiments demonstrate that our method is approximately twice as fast in both encoding and decoding compared to the parallelizable checkerboard context model, and it also achieves a 1.2% improvement in R-D performance compared to state-of-the-art learned image compression schemes. Our method also outperforms classical image codecs including H.266/VVC-intra (4:4:4) and some recent learned methods in rate-distortion performance, as validated by both PSNR and MS-SSIM metrics on the Kodak dataset.

stat.AP↗

arXiv2024

Learned Image Compression with Gaussian-Laplacian-Logistic Mixture Model and Concatenated Residual Modules

Recently deep learning-based image compression methods have achieved significant achievements and gradually outperformed traditional approaches including the latest standard Versatile Video Coding (VVC) in both PSNR and MS-SSIM metrics. Two key components of learned image compression are the entropy model of the latent representations and the encoding/decoding network architectures. Various models have been proposed, such as autoregressive, softmax, logistic mixture, Gaussian mixture, and Laplacian. Existing schemes only use one of these models. However, due to the vast diversity of images, it is not optimal to use one model for all images, even different regions within one image. In this paper, we propose a more flexible discretized Gaussian-Laplacian-Logistic mixture model (GLLMM) for the latent representations, which can adapt to different contents in different images and different regions of one image more accurately and efficiently, given the same complexity. Besides, in the encoding/decoding network design part, we propose a concatenated residual blocks (CRB), where multiple residual blocks are serially connected with additional shortcut connections. The CRB can improve the learning ability of the network, which can further improve the compression performance. Experimental results using the Kodak, Tecnick-100 and Tecnick-40 datasets show that the proposed scheme outperforms all the leading learning-based methods and existing compression standards including VVC intra coding (4:4:4 and 4:2:0) in terms of the PSNR and MS-SSIM. The source code is available at \url{https://github.com/fengyurenpingsheng}

eess.IV↗

arXiv2023

Guided Image Restoration via Simultaneous Feature and Image Guided Fusion

Guided image restoration (GIR), such as guided depth map super-resolution and pan-sharpening, aims to enhance a target image using guidance information from another image of the same scene. Currently, joint image filtering-inspired deep learning-based methods represent the state-of-the-art for GIR tasks. Those methods either deal with GIR in an end-to-end way by elaborately designing filtering-oriented deep neural network (DNN) modules, focusing on the feature-level fusion of inputs; or explicitly making use of the traditional joint filtering mechanism by parameterizing filtering coefficients with DNNs, working on image-level fusion. The former ones are good at recovering contextual information but tend to lose fine-grained details, while the latter ones can better retain textual information but might lead to content distortions. In this work, to inherit the advantages of both methodologies while mitigating their limitations, we proposed a Simultaneous Feature and Image Guided Fusion (SFIGF) network, that simultaneously considers feature and image-level guided fusion following the guided filter (GF) mechanism. In the feature domain, we connect the cross-attention (CA) with GF, and propose a GF-inspired CA module for better feature-level fusion; in the image domain, we fully explore the GF mechanism and design GF-like structure for better image-level fusion. Since guided fusion is implemented in both feature and image domains, the proposed SFIGF is expected to faithfully reconstruct both contextual and textual information from sources and thus lead to better GIR results. We apply SFIGF to 4 typical GIR tasks, and experimental results on these tasks demonstrate its effectiveness and general availability.

cs.CV↗

arXiv2023

MRGazer: Decoding Eye Gaze Points from Functional Magnetic Resonance Imaging in Individual Space

Eye-tracking research has proven valuable in understanding numerous cognitive functions. Recently, Frey et al. provided an exciting deep learning method for learning eye movements from fMRI data. However, it needed to co-register fMRI into standard space to obtain eyeballs masks, and thus required additional templates and was time consuming. To resolve this issue, in this paper, we propose a framework named MRGazer for predicting eye gaze points from fMRI in individual space. The MRGazer consisted of eyeballs extraction module and a residual network-based eye gaze prediction. Compared to the previous method, the proposed framework skips the fMRI co-registration step, simplifies the processing protocol and achieves end-to-end eye gaze regression. The proposed method achieved superior performance in a variety of eye movement tasks than the co-registration-based method, and delivered objective results within a shorter time (~ 0.02 Seconds for each volume) than prior method (~0.3 Seconds for each volume).

cs.CV↗

arXiv2023

Combined Experimental and Theoretical Studies on Iodine Capture of Zr-based Metal-Organic Frameworks: Effect of N-functionalization and Adsorption Mechanism

The potential leakage of nuclear waste, especially radioiodine, is a major safety concerning issue around the world. To remove radioiodine from nuclear waste efficiently, there is an urgent demand for adsorbents that possess both high stability and strong adsorption affinity for environmental remediation. Herein, two Zr-based metal-organic frameworks (Zr-MOFs) and their N-functionalized analogues have been synthesized and researched for iodine adsorption in both vapours and solutions. It was found that Zr-MOFs with N-enriched ligands (e.g., pyridine and amino) exhibited the faster iodine adsorption rate and the higher iodine uptake amount (e.g., reaching adsorption equilibrium within 4 hours with the removal rate of above 85% for iodine solution adsorption) than their unfunctionalized counterparts (UiO-66 and UiO-67). The critical role played by N-enriched groups in enhancing iodine adsorption has been revealed through versatile model fittings, X-ray photoelectron spectroscopy (XPS) and Raman spectroscopy characterizations, as well as density functional theory (DFT) calculations. Compared to those in amino-group, the N-atoms in pyridine-groups showed a deeper affinity towards iodine molecules. Remarkably, the N-enriched UiOs adsorbents also exhibited good recyclability, especially UiO-66-PYDC and UiO-67-NH2 could maintain the removal efficiency of 89.05% and 85.49% after four adsorption-desorption recycling tests. With the strong iodine uptake affinity and outstanding regeneration performance, this work has systematically investigated the impact of N-functionalization on the enhanced performance for iodine capture by using the N-enriched UiO MOFs as promising adsorbents, providing an insightful guideline into the physical chemistry of adsorption mechanism behind the radioiodine capture.

cond-mat.mtrl-sci↗

arXiv2023

Fast and High-Performance Learned Image Compression With Improved Checkerboard Context Model, Deformable Residual Module, and Knowledge Distillation

Deep learning-based image compression has made great progresses recently. However, many leading schemes use serial context-adaptive entropy model to improve the rate-distortion (R-D) performance, which is very slow. In addition, the complexities of the encoding and decoding networks are quite high and not suitable for many practical applications. In this paper, we introduce four techniques to balance the trade-off between the complexity and performance. We are the first to introduce deformable convolutional module in compression framework, which can remove more redundancies in the input image, thereby enhancing compression performance. Second, we design a checkerboard context model with two separate distribution parameter estimation networks and different probability models, which enables parallel decoding without sacrificing the performance compared to the sequential context-adaptive model. Third, we develop an improved three-step knowledge distillation and training scheme to achieve different trade-offs between the complexity and the performance of the decoder network, which transfers both the final and intermediate results of the teacher network to the student network to help its training. Fourth, we introduce $L_{1}$ regularization to make the numerical values of the latent representation more sparse. Then we only encode non-zero channels in the encoding and decoding process, which can greatly reduce the encoding and decoding time. Experiments show that compared to the state-of-the-art learned image coding scheme, our method can be about 20 times faster in encoding and 70-90 times faster in decoding, and our R-D performance is also $2.3 \%$ higher. Our method outperforms the traditional approach in H.266/VVC-intra (4:4:4) and some leading learned schemes in terms of PSNR and MS-SSIM metrics when testing on Kodak and Tecnick-40 datasets.

eess.IV↗

arXiv2023

Enhanced Residual SwinV2 Transformer for Learned Image Compression

Recently, the deep learning technology has been successfully applied in the field of image compression, leading to superior rate-distortion performance. However, a challenge of many learning-based approaches is that they often achieve better performance via sacrificing complexity, which making practical deployment difficult. To alleviate this issue, in this paper, we propose an effective and efficient learned image compression framework based on an enhanced residual Swinv2 transformer. To enhance the nonlinear representation of images in our framework, we use a feature enhancement module that consists of three consecutive convolutional layers. In the subsequent coding and hyper coding steps, we utilize a SwinV2 transformer-based attention mechanism to process the input image. The SwinV2 model can help to reduce model complexity while maintaining high performance. Experimental results show that the proposed method achieves comparable performance compared to some recent learned image compression methods on Kodak and Tecnick datasets, and outperforms some traditional codecs including VVC. In particular, our method achieves comparable results while reducing model complexity by 56% compared to these recent methods.

eess.IV↗

arXiv2023

ROI-based Deep Image Compression with Swin Transformers

Encoding the Region Of Interest (ROI) with better quality than the background has many applications including video conferencing systems, video surveillance and object-oriented vision tasks. In this paper, we propose a ROI-based image compression framework with Swin transformers as main building blocks for the autoencoder network. The binary ROI mask is integrated into different layers of the network to provide spatial information guidance. Based on the ROI mask, we can control the relative importance of the ROI and non-ROI by modifying the corresponding Lagrange multiplier $ λ$ for different regions. Experimental results show our model achieves higher ROI PSNR than other methods and modest average PSNR for human evaluation. When tested on models pre-trained with original images, it has superior object detection and instance segmentation performance on the COCO validation dataset.

cs.CV↗

arXiv2023

ClinicLens: Visual Analytics for Exploring and Optimizing the Testing Capacity of Clinics given Uncertainty

Clinic testing plays a critical role in containing infectious diseases such as COVID-19. However, one of the key research questions in fighting such pandemics is how to optimize testing capacities across clinics. In particular, domain experts expect to know exactly how to adjust the features that may affect testing capacities, given that dynamics and uncertainty make this a highly challenging problem. Hence, as a tool to support both policymakers and clinicians, we collaborated with domain experts to build ClinicLens, an interactive visual analytics system for exploring and optimizing the testing capacities of clinics. ClinicLens houses a range of features based on an aggregated set of COVID-19 data. It comprises Back-end Engine and Front-end Visualization that take users through an iterative exploration chain of extracting, training, and predicting testing-sensitive features and visual representations. It also combines AI4VIS and visual analytics to demonstrate how a clinic might optimize its testing capacity given the impacts of a range of features. Three qualitative case studies along with feedback from subject-matter experts validate that ClinicLens is both a useful and effective tool for exploring the trends in COVID-19 and optimizing clinic testing capacities across regions. The entire approach has been open-sourced online: https://github.com/YuDong5018/clinic-lens.

cs.HC↗

arXiv2023

Human Guided Ground-truth Generation for Realistic Image Super-resolution

How to generate the ground-truth (GT) image is a critical issue for training realistic image super-resolution (Real-ISR) models. Existing methods mostly take a set of high-resolution (HR) images as GTs and apply various degradations to simulate their low-resolution (LR) counterparts. Though great progress has been achieved, such an LR-HR pair generation scheme has several limitations. First, the perceptual quality of HR images may not be high enough, limiting the quality of Real-ISR outputs. Second, existing schemes do not consider much human perception in GT generation, and the trained models tend to produce over-smoothed results or unpleasant artifacts. With the above considerations, we propose a human guided GT generation scheme. We first elaborately train multiple image enhancement models to improve the perceptual quality of HR images, and enable one LR image having multiple HR counterparts. Human subjects are then involved to annotate the high quality regions among the enhanced HR images as GTs, and label the regions with unpleasant artifacts as negative samples. A human guided GT image dataset with both positive and negative samples is then constructed, and a loss function is proposed to train the Real-ISR models. Experiments show that the Real-ISR models trained on our dataset can produce perceptually more realistic results with less artifacts. Dataset and codes can be found at https://github.com/ChrisDud0257/HGGT

cs.CV↗

arXiv2023

SeqMapPDB: A Standalone Pipeline to Identify Representative Structures of Protein Sequences and Mapping Residue Indices in Real-Time at Proteome Scale

Motivation: 3D structures of proteins provide rich information for understanding their biochemical roles. Identifying the representative protein structures for protein sequences is essential for analysis of proteins at proteome scale. However, there are technical difficulties in identifying the representative structure of a given protein sequence and providing accurate mapping of residue indices. Existing databases of mapping between structures and sequences are usually static that are not suitable for studying proteomes with frequent gene model revisions. They often do not provide reliable and consistent representative structures that maximizes sequence coverage. Furthermore, proteins isomers are usually not properly resolved. Results: To overcome these difficulties, we have developed a computational pipeline called SeqMapPDB to provide high-quality representative PDB structures of given sequences. It provides mapping to structures that fully cover the sequences when available, or to the set of partial non-overlapping structural domains that maximally cover the query sequence. The residue indices are accurate mapped and isomeric proteins are resolved. SeqMapPDB is efficient and can rapidly carry out proteome-wide mapping to the selected version of reference genomes in real-time. Furthermore, SeqMapPDB provides the flexibility of a stand-alone pipeline for large scale mapping of in-house sequence and structure data. Availability: Our method is available at https://bitbucket.org/lianglabuic/seqmappdb with GNU GPL license.

q-bio.GN↗

arXiv2022

Reactive Vortexes in a Naturally Activated Process: Non-Diffusive Rotational Fluxes at Transition State Uncovered by Persistent Homology

Dynamics of reaction coordinates during barrier-crossing are key to understand activated processes in complex systems such as proteins. The default assumption from Kramers physical intuition is that of a diffusion process. However, the dynamics of barrier-crossing in natural complex molecules are largely unexplored. Here we investigate the transition dynamics of alanine-dipeptide isomerization, the simplest complex system with a large number of non-reaction coordinates that can serve as an adequate thermal bath feeding energy into the reaction coordinates. We separate conformations along the time axis and construct the dynamic probability surface of reaction. We quantify its topological structure and rotational flux using persistent homology and differential form. Our results uncovered a region with strong reactive vortex in the configuration-time space, where the highest probability peak and the transition state ensemble are located. This reactive region contains strong rotational fluxes: Most reactive trajectories swirl multiple times around this region in the subspace of the two most-important reaction coordinates. Furthermore, the rotational fluxes result from cooperative movement along the isocommitter surfaces and orthogonal barrier-crossing. Overall, our findings offer a first glimpse into the reactive vortex regions that characterize the non-diffusive dynamics of barrier-crossing of a naturally occurring activation process.

physics.chem-ph↗

arXiv2022

Graded extensions in K[Q, sigma]

Let V be a total valuation ring of a division ring K, Q be the additive group of the rational numbers, Aut(K) be the group of automorphisms of K. Let sigma be a group homomorphism from Q to Aut(K). Let K[Q, sigma ] be the skew group ring of Q over K. In this paper, we classify graded extensions of V in K[Q, sigma] into two types and study the structure of them.

math.RA↗

arXiv2022

Asymmetric Learned Image Compression with Multi-Scale Residual Block, Importance Map, and Post-Quantization Filtering

Recently, deep learning-based image compression has made signifcant progresses, and has achieved better ratedistortion (R-D) performance than the latest traditional method, H.266/VVC, in both subjective metric and the more challenging objective metric. However, a major problem is that many leading learned schemes cannot maintain a good trade-off between performance and complexity. In this paper, we propose an effcient and effective image coding framework, which achieves similar R-D performance with lower complexity than the state of the art. First, we develop an improved multi-scale residual block (MSRB) that can expand the receptive feld and is easier to obtain global information. It can further capture and reduce the spatial correlation of the latent representations. Second, a more advanced importance map network is introduced to adaptively allocate bits to different regions of the image. Third, we apply a 2D post-quantization flter (PQF) to reduce the quantization error, motivated by the Sample Adaptive Offset (SAO) flter in video coding. Moreover, We fnd that the complexity of encoder and decoder have different effects on image compression performance. Based on this observation, we design an asymmetric paradigm, in which the encoder employs three stages of MSRBs to improve the learning capacity, whereas the decoder only needs one stage of MSRB to yield satisfactory reconstruction, thereby reducing the decoding complexity without sacrifcing performance. Experimental results show that compared to the state-of-the-art method, the encoding and decoding time of the proposed method are about 17 times faster, and the R-D performance is only reduced by less than 1% on both Kodak and Tecnick datasets, which is still better than H.266/VVC(4:4:4) and other recent learning-based methods. Our source code is publicly available at https://github.com/fengyurenpingsheng.

eess.IV↗

arXiv2022

Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution

Efficient and effective real-world image super-resolution (Real-ISR) is a challenging task due to the unknown complex degradation of real-world images and the limited computation resources in practical applications. Recent research on Real-ISR has achieved significant progress by modeling the image degradation space; however, these methods largely rely on heavy backbone networks and they are inflexible to handle images of different degradation levels. In this paper, we propose an efficient and effective degradation-adaptive super-resolution (DASR) network, whose parameters are adaptively specified by estimating the degradation of each input image. Specifically, a tiny regression network is employed to predict the degradation parameters of the input image, while several convolutional experts with the same topology are jointly optimized to specify the network parameters via a non-linear mixture of experts. The joint optimization of multiple experts and the degradation-adaptive pipeline significantly extend the model capacity to handle degradations of various levels, while the inference remains efficient since only one adaptively specified network is used for super-resolving the input image. Our extensive experiments demonstrate that the proposed DASR is not only much more effective than existing methods on handling real-world images with different degradation levels but also efficient for easy deployment. Codes, models and datasets are available at https://github.com/csjliang/DASR.

cs.CV↗

arXiv2022

Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution

Single image super-resolution (SISR) with generative adversarial networks (GAN) has recently attracted increasing attention due to its potentials to generate rich details. However, the training of GAN is unstable, and it often introduces many perceptually unpleasant artifacts along with the generated details. In this paper, we demonstrate that it is possible to train a GAN-based SISR model which can stably generate perceptually realistic details while inhibiting visual artifacts. Based on the observation that the local statistics (e.g., residual variance) of artifact areas are often different from the areas of perceptually friendly details, we develop a framework to discriminate between GAN-generated artifacts and realistic details, and consequently generate an artifact map to regularize and stabilize the model training process. Our proposed locally discriminative learning (LDL) method is simple yet effective, which can be easily plugged in off-the-shelf SISR methods and boost their performance. Experiments demonstrate that LDL outperforms the state-of-the-art GAN based SISR methods, achieving not only higher reconstruction accuracy but also superior perceptual quality on both synthetic and real-world datasets. Codes and models are available at https://github.com/csjliang/LDL.

eess.IV↗

arXiv2022

Bilateral series and Ramanujan's radial limits

Ramanujan's last letter to Hardy explored the asymptotic properties of modular forms, as well as those of certain interesting $q$-series which he called \emph{mock theta functions}. For his mock theta function $f(q)$, he claimed that as $q$ approaches an even order $2k$ root of unity $ζ$, \[\lim_{q\to ζ} \big(f(q) - (-1)^k (1-q)(1-q^3)(1-q^5)\cdots (1-2q + 2q^4 - \cdots)\big) = O(1),\] and hinted at the existence of similar statements for his other mock theta functions. Recent work of Folsom-Ono-Rhoades provides a closed formula for the implied constant in this radial limit of $f(q)$. Here, by different methods, we prove similar results for all of Ramanujan's 5th order mock theta functions. Namely, we show that each 5th order mock theta function may be related to a modular bilateral series, and exploit this connection to obtain our results. We further explore other mock theta functions to which this method can be applied.

math.NT↗

arXiv2022

Auto-Weighted Layer Representation Based View Synthesis Distortion Estimation for 3-D Video Coding

Recently, various view synthesis distortion estimation models have been studied to better serve for 3-D video coding. However, they can hardly model the relationship quantitatively among different levels of depth changes, texture degeneration, and the view synthesis distortion (VSD), which is crucial for rate-distortion optimization and rate allocation. In this paper, an auto-weighted layer representation based view synthesis distortion estimation model is developed. Firstly, the sub-VSD (S-VSD) is defined according to the level of depth changes and their associated texture degeneration. After that, a set of theoretical derivations demonstrate that the VSD can be approximately decomposed into the S-VSDs multiplied by their associated weights. To obtain the S-VSDs, a layer-based representation of S-VSD is developed, where all the pixels with the same level of depth changes are represented with a layer to enable efficient S-VSD calculation at the layer level. Meanwhile, a nonlinear mapping function is learnt to accurately represent the relationship between the VSD and S-VSDs, automatically providing weights for S-VSDs during the VSD estimation. To learn such function, a dataset of VSD and its associated S-VSDs are built. Experimental results show that the VSD can be accurately estimated with the weights learnt by the nonlinear mapping function once its associated S-VSDs are available. The proposed method outperforms the relevant state-of-the-art methods in both accuracy and efficiency. The dataset and source code of the proposed method will be available at https://github.com/jianjin008/.

eess.IV↗