arXiv subjects

Jinming Liu

Publications and source records attributed to Jinming Liu.

At least 37 records · Page 2Linked to original sources

arXiv2024

Rate-Distortion-Cognition Controllable Versatile Neural Image Compression

Recently, the field of Image Coding for Machines (ICM) has garnered heightened interest and significant advances thanks to the rapid progress of learning-based techniques for image compression and analysis. Previous studies often require training separate codecs to support various bitrate levels, machine tasks, and networks, thus lacking both flexibility and practicality. To address these challenges, we propose a rate-distortion-cognition controllable versatile image compression, which method allows the users to adjust the bitrate (i.e., Rate), image reconstruction quality (i.e., Distortion), and machine task accuracy (i.e., Cognition) with a single neural model, achieving ultra-controllability. Specifically, we first introduce a cognition-oriented loss in the primary compression branch to train a codec for diverse machine tasks. This branch attains variable bitrate by regulating quantization degree through the latent code channels. To further enhance the quality of the reconstructed images, we employ an auxiliary branch to supplement residual information with a scalable bitstream. Ultimately, two branches use a `$βx + (1 - β) y$' interpolation strategy to achieve a balanced cognition-distortion trade-off. Extensive experiments demonstrate that our method yields satisfactory ICM performance and flexible Rate-Distortion-Cognition controlling.

cs.CV↗

arXiv2024

One at a Time: Progressive Multi-step Volumetric Probability Learning for Reliable 3D Scene Perception

Numerous studies have investigated the pivotal role of reliable 3D volume representation in scene perception tasks, such as multi-view stereo (MVS) and semantic scene completion (SSC). They typically construct 3D probability volumes directly with geometric correspondence, attempting to fully address the scene perception tasks in a single forward pass. However, such a single-step solution makes it hard to learn accurate and convincing volumetric probability, especially in challenging regions like unexpected occlusions and complicated light reflections. Therefore, this paper proposes to decompose the complicated 3D volume representation learning into a sequence of generative steps to facilitate fine and reliable scene perception. Considering the recent advances achieved by strong generative diffusion models, we introduce a multi-step learning framework, dubbed as VPD, dedicated to progressively refining the Volumetric Probability in a Diffusion process. Extensive experiments are conducted on scene perception tasks including multi-view stereo (MVS) and semantic scene completion (SSC), to validate the efficacy of our method in learning reliable volumetric representations. Notably, for the SSC task, our work stands out as the first to surpass LiDAR-based methods on the SemanticKITTI dataset.

cs.CV↗

arXiv2023

EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model

Although previous co-speech gesture generation methods are able to synthesize motions in line with speech content, it is still not enough to handle diverse and complicated motion distribution. The key challenges are: 1) the one-to-many nature between the speech content and gestures; 2) the correlation modeling between the body joints. In this paper, we present a novel framework (EMoG) to tackle the above challenges with denoising diffusion models: 1) To alleviate the one-to-many problem, we incorporate emotion clues to guide the generation process, making the generation much easier; 2) To model joint correlation, we propose to decompose the difficult gesture generation into two sub-problems: joint correlation modeling and temporal dynamics modeling. Then, the two sub-problems are explicitly tackled with our proposed Joint Correlation-aware transFormer (JCFormer). Through extensive evaluations, we demonstrate that our proposed method surpasses previous state-of-the-art approaches, offering substantial superiority in gesture synthesis.

cs.CV↗

arXiv2023

Observation and enhancement of room temperature bilinear magnetoelectric resistance in sputtered topological semimetal Pt3Sn

Topological semimetal materials have become a research hotspot due to their intrinsic strong spin-orbit coupling which leads to large charge-to-spin conversion efficiency and novel transport behaviors. In this work, we have observed a bilinear magnetoelectric resistance (BMER) of up to 0.1 nm2A-1Oe-1 in a singlelayer of sputtered semimetal Pt3Sn at room temperature. Different from previous observations, the value of BMER in sputtered Pt3Sn does not change out-of-plane due to the polycrystalline nature of Pt3Sn. The observation of BMER provides strong evidence of the existence of spin-momentum locking in the sputtered polycrystalline Pt3Sn. By adding an adjacent CoFeB magnetic layer, the BMER value of this bilayer system is doubled compared to the single Pt3Sn layer. This work broadens the material system in BMER study, which paves the way for the characterization of topological states and applications for spin memory and logic devices.

cond-mat.mtrl-sci↗

arXiv2023

Prompt-ICM: A Unified Framework towards Image Coding for Machines with Task-driven Prompts

Image coding for machines (ICM) aims to compress images to support downstream AI analysis instead of human perception. For ICM, developing a unified codec to reduce information redundancy while empowering the compressed features to support various vision tasks is very important, which inevitably faces two core challenges: 1) How should the compression strategy be adjusted based on the downstream tasks? 2) How to well adapt the compressed features to different downstream tasks? Inspired by recent advances in transferring large-scale pre-trained models to downstream tasks via prompting, in this work, we explore a new ICM framework, termed Prompt-ICM. To address both challenges by carefully learning task-driven prompts to coordinate well the compression process and downstream analysis. Specifically, our method is composed of two core designs: a) compression prompts, which are implemented as importance maps predicted by an information selector, and used to achieve different content-weighted bit allocations during compression according to different downstream tasks; b) task-adaptive prompts, which are instantiated as a few learnable parameters specifically for tuning compressed features for the specific intelligent task. Extensive experiments demonstrate that with a single feature codec and a few extra parameters, our proposed framework could efficiently support different kinds of intelligent tasks with much higher coding efficiency.

cs.CV↗

arXiv2023

Inpaint Anything: Segment Anything Meets Image Inpainting

Modern image inpainting systems, despite the significant progress, often struggle with mask selection and holes filling. Based on Segment-Anything Model (SAM), we make the first attempt to the mask-free image inpainting and propose a new paradigm of ``clicking and filling'', which is named as Inpaint Anything (IA). The core idea behind IA is to combine the strengths of different models in order to build a very powerful and user-friendly pipeline for solving inpainting-related problems. IA supports three main features: (i) Remove Anything: users could click on an object and IA will remove it and smooth the ``hole'' with the context; (ii) Fill Anything: after certain objects removal, users could provide text-based prompts to IA, and then it will fill the hole with the corresponding generative content via driving AIGC models like Stable Diffusion; (iii) Replace Anything: with IA, users have another option to retain the click-selected object and replace the remaining background with the newly generated scenes. We are also very willing to help everyone share and promote new projects based on our Inpaint Anything (IA). Our codes are available at https://github.com/geekyutao/Inpaint-Anything.

cs.CV↗

arXiv2023

Learned Image Compression with Mixed Transformer-CNN Architectures

Learned image compression (LIC) methods have exhibited promising progress and superior rate-distortion performance compared with classical image compression standards. Most existing LIC methods are Convolutional Neural Networks-based (CNN-based) or Transformer-based, which have different advantages. Exploiting both advantages is a point worth exploring, which has two challenges: 1) how to effectively fuse the two methods? 2) how to achieve higher performance with a suitable complexity? In this paper, we propose an efficient parallel Transformer-CNN Mixture (TCM) block with a controllable complexity to incorporate the local modeling ability of CNN and the non-local modeling ability of transformers to improve the overall architecture of image compression models. Besides, inspired by the recent progress of entropy estimation models and attention modules, we propose a channel-wise entropy model with parameter-efficient swin-transformer-based attention (SWAtten) modules by using channel squeezing. Experimental results demonstrate our proposed method achieves state-of-the-art rate-distortion performances on three different resolution datasets (i.e., Kodak, Tecnick, CLIC Professional Validation) compared to existing LIC methods. The code is at https://github.com/jmliu206/LIC_TCM.

eess.IV↗

arXiv2023

Multistage Spatial Context Models for Learned Image Compression

Recent state-of-the-art Learned Image Compression methods feature spatial context models, achieving great rate-distortion improvements over hyperprior methods. However, the autoregressive context model requires serial decoding, limiting runtime performance. The Checkerboard context model allows parallel decoding at a cost of reduced RD performance. We present a series of multistage spatial context models allowing both fast decoding and better RD performance. We split the latent space into square patches and decode serially within each patch while different patches are decoded in parallel. The proposed method features a comparable decoding speed to Checkerboard while reaching the RD performance of Autoregressive and even also outperforming Autoregressive. Inside each patch, the decoding order must be carefully decided as a bad order negatively impacts performance; therefore, we also propose a decoding order optimization algorithm.

cs.CV↗

arXiv2022

Semantic Segmentation in Learned Compressed Domain

Most machine vision tasks (e.g., semantic segmentation) are based on images encoded and decoded by image compression algorithms (e.g., JPEG). However, these decoded images in the pixel domain introduce distortion, and they are optimized for human perception, making the performance of machine vision tasks suboptimal. In this paper, we propose a method based on the compressed domain to improve segmentation tasks. i) A dynamic and a static channel selection method are proposed to reduce the redundancy of compressed representations that are obtained by encoding. ii) Two different transform modules are explored and analyzed to help the compressed representation be transformed as the features in the segmentation network. The experimental results show that we can save up to 15.8\% bitrates compared with a state-of-the-art compressed domain-based work while saving up to about 83.6\% bitrates and 44.8\% inference time compared with the pixel domain-based method.

cs.CV↗

arXiv2022

Learned Lossless Image Compression With Combined Autoregressive Models And Attention Modules

Lossless image compression is an essential research field in image compression. Recently, learning-based image compression methods achieved impressive performance compared with traditional lossless methods, such as WebP, JPEG2000, and FLIF. However, there are still many impressive lossy compression methods that can be applied to lossless compression. Therefore, in this paper, we explore the methods widely used in lossy compression and apply them to lossless compression. Inspired by the impressive performance of the Gaussian mixture model (GMM) shown in lossy compression, we generate a lossless network architecture with GMM. Besides noticing the successful achievements of attention modules and autoregressive models, we propose to utilize attention modules and add an extra autoregressive model for raw images in our network architecture to boost the performance. Experimental results show that our approach outperforms most classical lossless compression methods and existing learning-based methods.

eess.IV↗

arXiv2022

Room temperature spin-orbit torque efficiency in sputtered low-temperature superconductor delta-TaN

In the course of searching for promising topological materials for applications in future topological electronics, we evaluated spin-orbit torques (SOTs) in high-quality sputtered $δ-$TaN/Co20Fe60B20 devices through spin-torque ferromagnetic resonance ST-FMR and spin pumping measurements. From the ST-FMR characterization we observed a significant linewidth modulation in the magnetic Co20Fe60B20 layer attributed to the charge-to-spin conversion generated from the $δ-$TaN layer. Remarkably, the spin-torque efficiency determined from ST-FMR and spin pumping measurements is as large as $Θ =$ 0.034 and 0.031, respectively. These values are over two times larger than for $α-$Ta, but almost five times lower than for $β-$Ta, which can be attributed to the low room temperature electrical resistivity $\sim 74μΩ$ cm in $δ-$TaN. A large spin diffusion length of at least $\sim8$ nm is estimated, which is comparable to the spin diffusion length in pure Ta. Comprehensive experimental analysis, together with density functional theory calculations, indicates that the origin of the pronounced SOT effect in $δ-$TaN can be mostly related to a significant contribution from the Berry curvature associated with the presence of a topically nontrivial electronic band structure in the vicinity of the Fermi level (EF). Through additional detailed theoretical analysis, we also found that an isostructural allotrope of the superconducting $δ-$TaN phase, the simple hexagonal structure, $θ-$TaN, has larger Berry curvature, and that, together with expected reasonable charge conductivity, it can also be a promising candidate for exploring a generation of spin-orbit torque magnetic random access memory as cheap, temperature stable, and highly efficient spin current sources.

cond-mat.mes-hall↗

arXiv2022

Memory-Efficient Learned Image Compression with Pruned Hyperprior Module

Learned Image Compression (LIC) gradually became more and more famous in these years. The hyperprior-module-based LIC models have achieved remarkable rate-distortion performance. However, the memory cost of these LIC models is too large to actually apply them to various devices, especially to portable or edge devices. The parameter scale is directly linked with memory cost. In our research, we found the hyperprior module is not only highly over-parameterized, but also its latent representation contains redundant information. Therefore, we propose a novel pruning method named ERHP in this paper to efficiently reduce the memory cost of hyperprior module, while improving the network performance. The experiments show our method is effective, reducing at least 22.6% parameters in the whole model while achieving better rate-distortion performance.

eess.IV↗

arXiv2022

Element Doping Enhanced Charge-to-Spin Conversion Efficiency in Amorphous PtSn4 Dirac Semimetal

Topological semimetals (TSs) are promising candidates for low-power spin-orbit torque (SOT) devices due to their large charge-to-spin conversion efficiency. Here, we investigated the charge-to-spin conversion efficiency of amorphous PtSn4 (5 nm)/CoFeB (2.5-12.5 nm) layered structures prepared by a magnetron sputtering method at room temperature. The charge-to-spin ratio of PtSn4/CoFeB bilayers was 0.08, characterized by a spin torque ferromagnetic resonance (ST-FMR) technique. This ratio can further increase to 0.14 by inducing dopants, like Al and CoSi, into PtSn4. The dopants can also decrease (Al doping) or increase (CoSi doping) the resistivity of PtSn4. The work proposed a way to enhance the spin-orbit coupling (SOC) in amorphous TSs with dopants.

cond-mat.mtrl-sci↗

arXiv2020

An Investigation of Commercial Iron Oxide Nanoparticles: Advanced Structural and Magnetic Properties Characterization

Magnetic nanoparticles (MNPs) have been extensively used as tiny heating sources in magnetic hyperthermia therapy, contrast agents in magnetic resonance imaging (MRI), tracers in magnetic particle imaging (MPI), carriers for drug/gene delivery, etc. There have emerged many magnetic nanoparticle/microbeads suppliers since the last decade, such as Ocean NanoTech, Nanoprobes, US Research Nanomaterials, Miltenyi Biotec, micromod Partikeltechnologie GmbH, and nanoComposix, etc. In this paper, we report the physical and magnetic characterizations on iron oxide nanoparticle products from Ocean NanoTech. Standard characterization tools such as Vibrating-Sample Magnetometer (VSM), X-Ray Diffraction (XRD), Dynamic Light Scattering (DLS), Transmission Electron Microscopy (TEM), and Zeta Potential Analyzer are used to provide magnetic nanoparticle customers and researchers with an overview of these iron oxide nanoparticle products. In addition, the dynamic magnetic responses of these iron oxide nanoparticles in aqueous solutions are investigated under low and high frequency alternating magnetic fields, giving a standardized operating procedure for characterizing the MNPs from Ocean NanoTech, thereby yielding the best of magnetic nanoparticles for different applications.

physics.app-ph↗

arXiv2020

Portable Magnetic Particle Spectrometer (MPS) for Future Rapid and Wash-free Bioassays

Nowadays, there is an increasing demand for more accessible routine diagnostics for patients with respect to high accuracy, ease of use, and low cost. However, the quantitative and high accuracy bioassays in large hospitals and laboratories usually require trained technicians and equipment that is both bulky and expensive. In addition, the multi-step bioassays and long turnaround time could severely affect the disease surveillance and control especially in pandemics such as influenza and COVID-19. In view of this, a portable, quantitative bioassay device will be valuable in regions with scarce medical resources and help relieve burden on local healthcare systems. Herein, we introduce the MagiCoil diagnostic device, an inexpensive, portable, quantitative and rapid bioassay platform based on magnetic particle spectrometer (MPS) technique. MPS detects the dynamic magnetic responses of magnetic nanoparticles (MNPs) and uses the harmonics from oscillating MNPs as metrics for sensitive and quantitative bioassays. This device does not require trained technicians to operate and employs a fully automatic, one-step, wash-free assay with user friendly smartphone interface. Using a streptavidin-biotin binding system as a model, we show that the detection limit of the current portable device for streptavidin is 64 nM (equal to 5.12 pmole). In addition, this MPS technique is very versatile and allows for the detection of different diseases just by changing the surface modifications on MNPs.

physics.med-ph↗

arXiv2020

A Multi-stream Convolutional Neural Network for Micro-expression Recognition Using Optical Flow and EVM

Micro-expression (ME) recognition plays a crucial role in a wide range of applications, particularly in public security and psychotherapy. Recently, traditional methods rely excessively on machine learning design and the recognition rate is not high enough for its practical application because of its short duration and low intensity. On the other hand, some methods based on deep learning also cannot get high accuracy due to problems such as the imbalance of databases. To address these problems, we design a multi-stream convolutional neural network (MSCNN) for ME recognition in this paper. Specifically, we employ EVM and optical flow to magnify and visualize subtle movement changes in MEs and extract the masks from the optical flow images. And then, we add the masks, optical flow images, and grayscale images into the MSCNN. After that, in order to overcome the imbalance of databases, we added a random over-sampler after the Dense Layer of the neural network. Finally, extensive experiments are conducted on two public ME databases: CASME II and SAMM. Compared with many recent state-of-the-art approaches, our method achieves more promising recognition results.

cs.CV↗

arXiv2020

Magnetic Immunoassays: A Review of Virus and Pathogen Detection Before and Amidst the Coronavirus Disease-19 (COVID-19)

The novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which causes coronavirus disease 2019 (COVID-19), is a threat to the global healthcare system and economic security. As of July 2020, no specific drugs or vaccines are yet available for COVID-19, fast and accurate diagnosis for SARS-CoV-2 is essential in slowing down the spread of COVID-19 and for efficient implementation of control and containment strategies. Magnetic immunoassay is a novel and emerging topic representing the frontiers of current biosensing and magnetics areas. The past decade has seen rapid growth in applying magnetic tools for biological and biomedical applications. Recent advances in magnetic materials and nanotechnologies have transformed current diagnostic methods to nanoscale and pushed the detection limit to early stage disease diagnosis. Herein, this review covers the literatures of magnetic immunoassay platforms for virus and pathogen detections, before COVID-19. We reviewed the popular magnetic immunoassay platforms including magnetoresistance (MR) sensors, magnetic particle spectroscopy (MPS), and nuclear magnetic resonance (NMR). Magnetic Point-of-Care (POC) diagnostic kits are also reviewed aiming at developing plug-and-play diagnostics to manage the SARS-CoV-2 outbreak as well as preventing future epidemics. In addition, other platforms that use magnetic materials as auxiliary tools for enhanced pathogen and virus detections are also covered. The goal of this review is to inform the researchers of diagnostic and surveillance platforms for SARS-CoV-2 and their performances.

physics.med-ph↗

arXiv2020

Magnetic Particle Spectroscopy: A Short Review of Applications

Magnetic particle spectroscopy (MPS), also called magnetization response spectroscopy, is a novel measurement tool derived from magnetic particle imaging (MPI). It can be interpreted as a zero-dimensional version of MPI scanner. MPS was primarily designed for characterizing superparamagnetic iron oxide nanoparticles (SPIONs) regarding their applicability for MPI. In recent years, it has evolved into an independent, versatile, highly sensitive, inexpensive platform for biological and biomedical assays, cell labeling and tracking, and blood analysis. MPS has also developed into an auxiliary tool for magnetic imaging and hyperthermia by providing high spatial and temporal mappings of temperature and viscosity. Furthermore, other MPS-based applications are being explored such as magnetic fingerprints for product tracking and identification in supply chains. There are a variety of novel MPS-based applications being reported and demonstrated by many groups. In this short review, we highlighted some of the representative applications based on MPS platform, thereby providing a roadmap of this technology and our insights for researchers in this area.

physics.app-ph↗