arXiv subjects

Zhao Wang

Publications and source records attributed to Zhao Wang.

At least 127 records · Page 7Linked to original sources

arXiv2024

Tunnel Try-on: Excavating Spatial-temporal Tunnels for High-quality Virtual Try-on in Videos

Video try-on is a challenging task and has not been well tackled in previous works. The main obstacle lies in preserving the details of the clothing and modeling the coherent motions simultaneously. Faced with those difficulties, we address video try-on by proposing a diffusion-based framework named "Tunnel Try-on." The core idea is excavating a "focus tunnel" in the input video that gives close-up shots around the clothing regions. We zoom in on the region in the tunnel to better preserve the fine details of the clothing. To generate coherent motions, we first leverage the Kalman filter to construct smooth crops in the focus tunnel and inject the position embedding of the tunnel into attention layers to improve the continuity of the generated videos. In addition, we develop an environment encoder to extract the context information outside the tunnels as supplementary cues. Equipped with these techniques, Tunnel Try-on keeps the fine details of the clothing and synthesizes stable and smooth videos. Demonstrating significant advancements, Tunnel Try-on could be regarded as the first attempt toward the commercial-level application of virtual try-on in videos.

cs.CV↗

arXiv2024

Siracusa: A 16 nm Heterogenous RISC-V SoC for Extended Reality with At-MRAM Neural Engine

Extended reality (XR) applications are Machine Learning (ML)-intensive, featuring deep neural networks (DNNs) with millions of weights, tightly latency-bound (10-20 ms end-to-end), and power-constrained (low tens of mW average power). While ML performance and efficiency can be achieved by introducing neural engines within low-power systems-on-chip (SoCs), system-level power for nontrivial DNNs depends strongly on the energy of non-volatile memory (NVM) access for network weights. This work introduces Siracusa, a near-sensor heterogeneous SoC for next-generation XR devices manufactured in 16 nm CMOS. Siracusa couples an octa-core cluster of RISC-V digital signal processing cores with a novel tightly-coupled "At-Memory" integration between a state-of-the-art digital neural engine called N-EUREKA and an on-chip NVM based on magnetoresistive memory(MRAM), achieving 1.7x higher throughput and 3x better energy efficiency than XR SoCs using NVM as background memory. The fabricated SoC prototype achieves an area efficiency of 65.2 GOp/s/mm2 and a peak energy efficiency of 8.84 TOp/J for DNN inference while supporting complex heterogeneous application workloads, which combine ML with conventional signal processing and control.

cs.AR↗

arXiv2024

Existence Is Chaos: Enhancing 3D Human Motion Prediction with Uncertainty Consideration

Human motion prediction is consisting in forecasting future body poses from historically observed sequences. It is a longstanding challenge due to motion's complex dynamics and uncertainty. Existing methods focus on building up complicated neural networks to model the motion dynamics. The predicted results are required to be strictly similar to the training samples with L2 loss in current training pipeline. However, little attention has been paid to the uncertainty property which is crucial to the prediction task. We argue that the recorded motion in training data could be an observation of possible future, rather than a predetermined result. In addition, existing works calculate the predicted error on each future frame equally during training, while recent work indicated that different frames could play different roles. In this work, a novel computationally efficient encoder-decoder model with uncertainty consideration is proposed, which could learn proper characteristics for future frames by a dynamic function. Experimental results on benchmark datasets demonstrate that our uncertainty consideration approach has obvious advantages both in quantity and quality. Moreover, the proposed method could produce motion sequences with much better quality that avoids the intractable shaking artefacts. We believe our work could provide a novel perspective to consider the uncertainty quality for the general motion prediction task and encourage the studies in this field. The code will be available in https://github.com/Motionpre/Adaptive-Salient-Loss-SAGGB.

cs.CV↗

arXiv2024

An AI-Driven Approach to Wind Turbine Bearing Fault Diagnosis from Acoustic Signals

This study aimed to develop a deep learning model for the classification of bearing faults in wind turbine generators from acoustic signals. A convolutional LSTM model was successfully constructed and trained by using audio data from five predefined fault types for both training and validation. To create the dataset, raw audio signal data was collected and processed in frames to capture time and frequency domain information. The model exhibited outstanding accuracy on training samples and demonstrated excellent generalization ability during validation, indicating its proficiency of generalization capability. On the test samples, the model achieved remarkable classification performance, with an overall accuracy exceeding 99.5%, and a false positive rate of less than 1% for normal status. The findings of this study provide essential support for the diagnosis and maintenance of bearing faults in wind turbine generators, with the potential to enhance the reliability and efficiency of wind power generation.

cs.SD↗

arXiv2024

Efficient Transferability Assessment for Selection of Pre-trained Detectors

Large-scale pre-training followed by downstream fine-tuning is an effective solution for transferring deep-learning-based models. Since finetuning all possible pre-trained models is computational costly, we aim to predict the transferability performance of these pre-trained models in a computational efficient manner. Different from previous work that seek out suitable models for downstream classification and segmentation tasks, this paper studies the efficient transferability assessment of pre-trained object detectors. To this end, we build up a detector transferability benchmark which contains a large and diverse zoo of pre-trained detectors with various architectures, source datasets and training schemes. Given this zoo, we adopt 7 target datasets from 5 diverse domains as the downstream target tasks for evaluation. Further, we propose to assess classification and regression sub-tasks simultaneously in a unified framework. Additionally, we design a complementary metric for evaluating tasks with varying objects. Experimental results demonstrate that our method outperforms other state-of-the-art approaches in assessing transferability under different target domains while efficiently reducing wall-clock time 32$\times$ and requires a mere 5.2\% memory footprint compared to brute-force fine-tuning of all pre-trained detectors.

cs.CV↗

arXiv2024

Open-Vocabulary Object Detection with Meta Prompt Representation and Instance Contrastive Optimization

Classical object detectors are incapable of detecting novel class objects that are not encountered before. Regarding this issue, Open-Vocabulary Object Detection (OVOD) is proposed, which aims to detect the objects in the candidate class list. However, current OVOD models are suffering from overfitting on the base classes, heavily relying on the large-scale extra data, and complex training process. To overcome these issues, we propose a novel framework with Meta prompt and Instance Contrastive learning (MIC) schemes. Firstly, we simulate a novel-class-emerging scenario to help the prompt learner that learns class and background prompts generalize to novel classes. Secondly, we design an instance-level contrastive strategy to promote intra-class compactness and inter-class separation, which benefits generalization of the detector to novel class objects. Without using knowledge distillation, ensemble model or extra training data during detector training, our proposed MIC outperforms previous SOTA methods trained with these complex techniques on LVIS. Most importantly, MIC shows great generalization ability on novel classes, e.g., with $+4.3\%$ and $+1.9\% \ \mathrm{AP}$ improvement compared with previous SOTA on COCO and Objects365, respectively.

cs.CV↗

arXiv2024

On the optimal control problem for the k-FORQ/MCH equation with viscosity

In this paper, we investigate the optimal control problem for the k-FORQ/MCH equation with strong viscosity.We prove the existence and uniqueness of this equation under the initial and boundary conditions by Galerkin method. From these results and the Lion's theory we deduce the existence of an optimal solution to the control problem governed by the viscous k-FORQ/MCH equation. Using augmented Lagrangian method, we deduce the first-order necessary optimality condition and two second-order sufficient optimality conditions.

math.OC↗

arXiv2024

Perceptual Video Coding for Machines via Satisfied Machine Ratio Modeling

Video Coding for Machines (VCM) aims to compress visual signals for machine analysis. However, existing methods only consider a few machines, neglecting the majority. Moreover, the machine's perceptual characteristics are not leveraged effectively, resulting in suboptimal compression efficiency. To overcome these limitations, this paper introduces Satisfied Machine Ratio (SMR), a metric that statistically evaluates the perceptual quality of compressed images and videos for machines by aggregating satisfaction scores from them. Each score is derived from machine perceptual differences between original and compressed images. Targeting image classification and object detection tasks, we build two representative machine libraries for SMR annotation and create a large-scale SMR dataset to facilitate SMR studies. We then propose an SMR prediction model based on the correlation between deep feature differences and SMR. Furthermore, we introduce an auxiliary task to increase the prediction accuracy by predicting the SMR difference between two images in different quality. Extensive experiments demonstrate that SMR models significantly improve compression performance for machines and exhibit robust generalizability on unseen machines, codecs, datasets, and frame types. SMR enables perceptual coding for machines and propels VCM from specificity to generality. Code is available at https://github.com/ywwynm/SMR.

cs.CV↗

arXiv2024

Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train

Foundation models have exhibited remarkable success in various applications, such as disease diagnosis and text report generation. To date, a foundation model for endoscopic video analysis is still lacking. In this paper, we propose Endo-FM, a foundation model specifically developed using massive endoscopic video data. First, we build a video transformer, which captures both local and global long-range dependencies across spatial and temporal dimensions. Second, we pre-train our transformer model using global and local views via a self-supervised manner, aiming to make it robust to spatial-temporal variations and discriminative across different scenes. To develop the foundation model, we construct a large-scale endoscopy video dataset by combining 9 publicly available datasets and a privately collected dataset from Baoshan Branch of Renji Hospital in Shanghai, China. Our dataset overall consists of over 33K video clips with up to 5 million frames, encompassing various protocols, target organs, and disease types. Our pre-trained Endo-FM can be easily adopted for a given downstream task via fine-tuning by serving as the backbone. With experiments on 3 different types of downstream tasks, including classification, segmentation, and detection, our Endo-FM surpasses the current state-of-the-art (SOTA) self-supervised pre-training and adapter-based transfer learning methods by a significant margin, such as VCL (3.1% F1, 4.8% Dice, and 5.5% F1 for classification, segmentation, and detection) and ST-Adapter (5.9% F1, 9.6% Dice, and 9.9% F1 for classification, segmentation, and detection). Code, datasets, and models are released at https://github.com/med-air/Endo-FM.

cs.CV↗

arXiv2023

A graph-based multimodal framework to predict gentrification

Gentrification--the transformation of a low-income urban area caused by the influx of affluent residents--has many revitalizing benefits. However, it also poses extremely concerning challenges to low-income residents. To help policymakers take targeted and early action in protecting low-income residents, researchers have recently proposed several machine learning models to predict gentrification using socioeconomic and image features. Building upon previous studies, we propose a novel graph-based multimodal deep learning framework to predict gentrification based on urban networks of tracts and essential facilities (e.g., schools, hospitals, and subway stations). We train and test the proposed framework using data from Chicago, New York City, and Los Angeles. The model successfully predicts census-tract level gentrification with 0.9 precision on average. Moreover, the framework discovers a previously unexamined strong relationship between schools and gentrification, which provides a basis for further exploration of social factors affecting gentrification.

cs.CY↗

arXiv2023

Incoherent illumination for motion-based imaging through thick scattering medium

Object-motion-based speckle correlation can recover hidden objects from any inhomogeneous medium, which takes advantage of the inherent connection that the cross-correlation between speckle patterns can reflect the autocorrelation of object, providing a route for imaging through or inside thick scattering media. However, once the object is phase-modulated, the above-mentioned relation will not be satisfied under coherent illumination, and the objects cannot be recovered using the existing approaches. Here, we propose an incoherent illumination method for object-motion-based imaging. Theoretical analysis and experimental results show that the cross-correlation between the object-motion-based speckle patterns can be directly used to represent the intensity autocorrelation of the object, making it possible to recover hidden objects regardless of whether the object is phase-modulated or not. Moreover, the proposed approach has a lower root-mean-square error for extracting the autocorrelation patterns of the hidden object. The proposed imaging mechanism blazes a way of imaging moving objects with scattering-induced or intrinsic phase profile, which is in favor of complex imaging scenarios such as inhomogeneous object imaging, deep tissue imaging, and passive lighting scattering imaging.

physics.optics↗

arXiv2023

From Ad-Hoc to Systematic: A Strategy for Imposing General Boundary Conditions in Discretized PDEs in variational quantum algorithm

We proposed a general quantum-computing-based algorithm that harnesses the exponential power of noisy intermediate-scale quantum (NISQ) devices in solving partial differential equations (PDE). This variational quantum eigensolver (VQE)-inspired approach transcends previous idealized model demonstrations constrained by strict and simplistic boundary conditions. It enables the imposition of arbitrary boundary conditions, significantly expanding its potential and adaptability for real-world applications, achieving this "from ad-hoc to systematic" concept. We have implemented this method using the fourth-order PDE (the Euler-Bernoulli beam) as example and showcased its effectiveness with four different boundary conditions. This framework enables expectation evaluations independent of problem size, harnessing the exponentially growing state space inherent in quantum computing, resulting in exceptional scalability. This method paves the way for applying quantum computing to practical engineering applications.

quant-ph↗

arXiv2023

Orbital-angular-momentum dependent speckles for spatial mode sorting and multiplexed data transmission

Characterizing the orbital angular momentum (OAM) of a vortex beam is critically important for OAM-encoded data transfer. However, in typical OAM-based applications where vortex beams transmit through diffusers, the accompanying scattering effect tends to be either deliberately prevented, or characterized and then modulated actively based on complex wavefront shaping and interferometry techniques. Here, we aim to investigate the characteristics of blurred speckles obtained after a vortex beam transmits through a ground glass diffuser. It is theoretically and experimentally demonstrated that a cross-correlation annulus can be identified by implementing the cross-correlation operation between speckle patterns corresponding to vortex beams with different OAM values. Besides, it is worth noting that, the size of the cross-correlation annulus is determined by the absolute value of the topological charge difference between the two corresponding vortex beams. Based on this mechanism, the OAM modes can be easily sorted from the incoherently measured OAM-dependent speckles as well as their cross-correlation. Furthermore, to make full use of the orthogonal feature of the OAM-dependent speckles, demultiplexing of OAM-encoded data transfer is verified using a ground glass diffuser. Both 8-bit grayscale and 24-bit RGB OAM-encoded data transfers are carried out in experiments with superior error rates. We can conclude that the OAM-dependent speckles can be not only utilized as a competitive candidate for the OAM mode sorting function in a simple way but also provide an efficient method for the demultiplexing of OAM-encoded data transfer in a practical application.

physics.optics↗

arXiv2023

Evolution of Fullerenes in Circumstellar Envelopes by Carbon Condensation: Insights from Reactive Molecular Dynamics Simulations

Fullerenes, including \ce{C60} and \ce{C70}, have been detected in various astronomical environments. Understanding how their structures evolve over time is essential for gaining insights into their life cycle and making further observations. To address this, we conducted reactive molecular dynamics simulations to investigate the evolution of fullerenes in the circumstellar envelopes surrounding carbon-rich asymptotic giant branch stars. Our simulations employed a bottom-up chemistry scheme, wherein fullerenes grow by absorbing and condensing small carbon-based molecules. The results revealed the formation of different structures through heterogeneous reactions based on hydrogen concentration, leading to the emergence of onion-like nanostructures or single-layer fullerenes. To examine the impact of these structural changes on the infrared emission characteristics of fullerenes, we performed quantum chemical calculations. The results indicate that as fullerenes grow larger, additional emission features are introduced in the infrared spectrum. Moreover, two-layered fullerenes show noticeable blue-shift or weakening effects on the bands associated with out-of-plane vibration modes.

astro-ph.SR↗

arXiv2023

Density Functional Theory Calculations on the Interstellar Formation of Biomolecules

The density functional theory (DFT) is the most versatile electronic structure method used in quantum chemical calculations, and is increasingly applied in astrochemical research. This mini-review provides an overview of the applications of DFT calculations in understanding the chemistry that occurs in star-forming regions. We survey investigations into the formation of biologically-relevant compounds such as nucleobases in the interstellar medium, and also covers the formation of both achiral and chiral amino acids, as well as biologically-relevant molecules such as sugars, and nitrogen-containing polycyclic aromatic hydrocarbons. Additionally, DFT calculations are used to estimate the potential barriers for chemical reactions in astronomical environments. We conclude by noting several areas that require more research, such as the formation pathways of chiral amino acids, complex sugars and other biologically-important molecules, and the role of environmental factors in the formation of interstellar biomolecules.

astro-ph.GA↗

arXiv2023

Charge equilibration of Laser-accelerated Carbon Ions in Foam Target

The charge equilibration of laser-accelerated carbon ion beams in 2 mg/cm3 foam target was investigated experimentally. The ions were generated through target normal sheath acceleration mechanism in laser-foil interaction scheme. This allows to get the equilibrium charge state in wide energy range near Bragg peak within a single shot. By using foam, the charge equilibration measurement in density regime between gas and solid state was firstly reached out experimentally. It was found that the theoretical predictions with tabulated cross section data for gas target greatly underestimated the charge states. The experimental data are in close agreement with both semi-empirical formula as well as rate equation predictions based on ion-solid interactions. The important role of target density effects that increase the ionization probability and decrease the electron capture probability through frequent multi-collisions in foam are demonstrated. The double electron processes are shown to have little influence on the average charge states. The findings are essential for high energy density physics research where the foams are widely used, and have impacts on a broad range of applications in medical, biological and material fields. The method also provides a new approach to investigate the interaction mechanism of swift heavy ions in matter by taking advantage of the laser-accelerated short-pulse wide-energy range ions.

physics.atom-ph↗

arXiv2023

Semantic Face Compression for Metaverse: A Compact 3D Descriptor Based Approach

In this letter, we envision a new metaverse communication paradigm for virtual avatar faces, and develop the semantic face compression with compact 3D facial descriptors. The fundamental principle is that the communication of virtual avatar faces primarily emphasizes the conveyance of semantic information. In light of this, the proposed scheme offers the advantages of being highly flexible, efficient and semantically meaningful. The semantic face compression, which allows the communication of the descriptors for artificial intelligence based understanding, could facilitate numerous applications without the involvement of humans in metaverse. The promise of the proposed paradigm is also demonstrated by performance comparisons with the state-of-the-art video coding standard, Versatile Video Coding. A significant improvement in terms of rate-accuracy performance has been achieved. The proposed scheme is expected to enable numerous applications, such as digital human communication based on machine analysis, and to form the cornerstone of interaction and communication in the metaverse.

cs.CV↗

arXiv2023

Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregation

In this paper, a novel Diffusion-based 3D Pose estimation (D3DP) method with Joint-wise reProjection-based Multi-hypothesis Aggregation (JPMA) is proposed for probabilistic 3D human pose estimation. On the one hand, D3DP generates multiple possible 3D pose hypotheses for a single 2D observation. It gradually diffuses the ground truth 3D poses to a random distribution, and learns a denoiser conditioned on 2D keypoints to recover the uncontaminated 3D poses. The proposed D3DP is compatible with existing 3D pose estimators and supports users to balance efficiency and accuracy during inference through two customizable parameters. On the other hand, JPMA is proposed to assemble multiple hypotheses generated by D3DP into a single 3D pose for practical use. It reprojects 3D pose hypotheses to the 2D camera plane, selects the best hypothesis joint-by-joint based on the reprojection errors, and combines the selected joints into the final pose. The proposed JPMA conducts aggregation at the joint level and makes use of the 2D prior information, both of which have been overlooked by previous approaches. Extensive experiments on Human3.6M and MPI-INF-3DHP datasets show that our method outperforms the state-of-the-art deterministic and probabilistic approaches by 1.5% and 8.9%, respectively. Code is available at https://github.com/paTRICK-swk/D3DP.

cs.CV↗