Searcharxiv⌕ Search

arXiv subjects

Jiaqi Yang

Publications and source records attributed to Jiaqi Yang.

At least 91 records · Page 5Linked to original sources
arXiv2024

Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from an unseen image, and then animate it with a reference video or audio to generate a talking portrait video. The existing methods fail to simultaneously achieve the goals of accurate 3D avatar reconstruction and stable talking face animation. Besides, while the existing works mainly focus on synthesizing the head part, it is also vital to generate natural torso and background segments to obtain a realistic talking portrait video. To address these limitations, we present Real3D-Potrait, a framework that (1) improves the one-shot 3D reconstruction power with a large image-to-plane model that distills 3D prior knowledge from a 3D face generative model; (2) facilitates accurate motion-conditioned animation with an efficient motion adapter; (3) synthesizes realistic video with natural torso movement and switchable background using a head-torso-background super-resolution model; and (4) supports one-shot audio-driven talking face generation with a generalizable audio-to-motion model. Extensive experiments show that Real3D-Portrait generalizes well to unseen identities and generates more realistic talking portrait videos compared to previous methods. Video samples and source code are available at https://real3dportrait.github.io .

cs.CV↗
arXiv2024

MV-ROPE: Multi-view Constraints for Robust Category-level Object Pose and Size Estimation

Recently there has been a growing interest in category-level object pose and size estimation, and prevailing methods commonly rely on single view RGB-D images. However, one disadvantage of such methods is that they require accurate depth maps which cannot be produced by consumer-grade sensors. Furthermore, many practical real-world situations involve a moving camera that continuously observes its surroundings, and the temporal information of the input video streams is simply overlooked by single-view methods. We propose a novel solution that makes use of RGB video streams. Our framework consists of three modules: a scale-aware monocular dense SLAM solution, a lightweight object pose predictor, and an object-level pose graph optimizer. The SLAM module utilizes a video stream and additional scale-sensitive readings to estimate camera poses and metric depth. The object pose predictor then generates canonical object representations from RGB images. The object pose is estimated through geometric registration of these canonical object representations with estimated object depth points. All per-view estimates finally undergo optimization within a pose graph, culminating in the output of robust and accurate canonical object poses. Our experimental results demonstrate that when utilizing public dataset sequences with high-quality depth information, the proposed method exhibits comparable performance to state-of-the-art RGB-D methods. We also collect and evaluate on new datasets containing depth maps of varying quality to further quantitatively benchmark the proposed method alongside previous RGB-D based methods. We demonstrate a significant advantage in scenarios where depth input is absent or the quality of depth sensing is limited.

cs.CV↗
arXiv2024

Instance by Instance: An Iterative Framework for Multi-instance 3D Registration

Multi-instance registration is a challenging problem in computer vision and robotics, where multiple instances of an object need to be registered in a standard coordinate system. In this work, we propose the first iterative framework called instance-by-instance (IBI) for multi-instance 3D registration (MI-3DReg). It successively registers all instances in a given scenario, starting from the easiest and progressing to more challenging ones. Throughout the iterative process, outliers are eliminated continuously, leading to an increasing inlier rate for the remaining and more challenging instances. Under the IBI framework, we further propose a sparse-to-dense-correspondence-based multi-instance registration method (IBI-S2DC) to achieve robust MI-3DReg. Experiments on the synthetic and real datasets have demonstrated the effectiveness of IBI and suggested the new state-of-the-art performance of IBI-S2DC, e.g., our MHF1 is 12.02%/12.35% higher than the existing state-of-the-art method ECC on the synthetic/real datasets.

cs.CV↗
arXiv2024

Relative Pose for Nonrigid Multi-Perspective Cameras: The Static Case

Multi-perspective cameras with potentially non-overlapping fields of view have become an important exteroceptive sensing modality in a number of applications such as intelligent vehicles, drones, and mixed reality headsets. In this work, we challenge one of the basic assumptions made in these scenarios, which is that the multi-camera rig is rigid. More specifically, we are considering the problem of estimating the relative pose between a static non-rigid rig in different spatial orientations while taking into account the effect of gravity onto the system. The deformable physical connections between each camera and the body center are approximated by a simple cantilever model, and inserted into the generalized epipolar constraint. Our results lead us to the important insight that the latent parameters of the deformation model, meaning the gravity vector in both views, become observable. We present a concise analysis of the observability of all variables based on noise, outliers, and rig rigidity for two different algorithms. The first one is a vision-only alternative, while the second one makes use of additional gravity measurements. To conclude, we demonstrate the ability to sense gravity in a real-world example, and discuss practical implications.

cs.RO↗
arXiv2023

Square Moiré Superlattices in Twisted Two-Dimensional Halide Perovskites

Moiré superlattices have emerged as a new platform for studying strongly correlated quantum phenomena, but these systems have been largely limited to van der Waals layer two-dimensional (2D) materials. Here we introduce moiré superlattices leveraging ultra-thin, ligand-free halide perovskites, facilitated by ionic interactions. Square moiré superlattices with varying periodic lengths are clearly visualized through high-resolution transmission electron microscopy. Twist-angle-dependent transient photoluminescence microscopy and electrical characterizations indicate the emergence of localized bright excitons and trapped charge carriers near a twist angle of ~10°. The localized excitons are accompanied by enhanced exciton emission, attributed to an increased oscillator strength by a theoretically forecasted flat band. This work illustrates the potential of extended ionic interaction in realizing moiré physics at room temperature, broadening the horizon for future investigations.

cond-mat.mtrl-sci↗
arXiv2023

Towards Accelerating Particle-Resolved Direct Numerical Simulation with Neural Operators

We present our ongoing work aimed at accelerating a particle-resolved direct numerical simulation model designed to study aerosol-cloud-turbulence interactions. The dynamical model consists of two main components - a set of fluid dynamics equations for air velocity, temperature, and humidity, coupled with a set of equations for particle (i.e., cloud droplet) tracing. Rather than attempting to replace the original numerical solution method in its entirety with a machine learning (ML) method, we consider developing a hybrid approach. We exploit the potential of neural operator learning to yield fast and accurate surrogate models and, in this study, develop such surrogates for the velocity and vorticity fields. We discuss results from numerical experiments designed to assess the performance of ML architectures under consideration as well as their suitability for capturing the behavior of relevant dynamical systems.

physics.flu-dyn↗
arXiv2023

Discovering Novel Halide Perovskite Alloys using Multi-Fidelity Machine Learning and Genetic Algorithm

Expanding the pool of stable halide perovskites with attractive optoelectronic properties is crucial to addressing current limitations in their performance as photovoltaic (PV) absorbers. In this article, we demonstrate how a high-throughput density functional theory (DFT) dataset of halide perovskite alloys can be used to train accurate surrogate models for property prediction and subsequently perform inverse design using genetic algorithm (GA). Our dataset consists of decomposition energies, band gaps, and photovoltaic efficiencies of nearly 800 pure and mixed composition ABX$_3$ compounds from both the GGA-PBE and HSE06 functionals, and are combined with ~ 100 experimental data points collected from the literature. Multi-fidelity random forest regression models are trained on the DFT + experimental dataset for each property using descriptors that one-hot encode composition, phase, and fidelity, and additionally include well-known elemental or molecular properties of species at the A, B, and X sites. Rigorously optimized models are deployed for experiment-level prediction over > 150,000 hypothetical compounds, leading to thousands of promising materials with low decomposition energy, band gap between 1 and 2 eV, and efficiency > 15%. Surrogate models are further combined with GA using an objective function to maintain chemical feasibility, minimize decomposition energy, maximize PV efficiency, and keep band gap between 1 and 2 eV; hundreds more optimal compositions and phases are thus discovered. We present an analysis of the screened and inverse-designed materials, visualize ternary phase diagrams generated for many systems of interest using ML predictions, and suggest strategies for further improvement and expansion in the future.

cond-mat.mtrl-sci↗
arXiv2023

First Principles Investigation of Polymorphism in Halide Perovskites

Halide perovskites have been extensively studied as materials of interest for optoelectronic applications. There is a major emphasis on ways to tailor the stability, defect behavior, electronic band structure, and optical absorption in halide perovskites, by changing the composition or structure. In this work, we present our contribution to this field in the form of a comprehensive computational investigation of properties as a function of the perovskite phase, different degrees of lattice strains and octahedral distortion and rotation, and the ordering of cations in perovskite alloys. We performed first principles-based density functional theory computations using multiple semi-local and non-local hybrid functionals to calculate optimized lattice parameters, energies of decomposition, electronic band gaps, and theoretical photovoltaic efficiencies. Trends and critical observations from the high-throughput dataset are discussed, especially in terms of the range of optoelectronic properties achievable while keeping the material in a (meta)stable phase or distorted, strained, or differently ordered polymorph. All data is made openly available to the community and is currently being utilized to train state-of-the-art machine learning models for accelerated prediction and discovery, as well as to guide rational experimental discovery.

cond-mat.mtrl-sci↗
arXiv2023

On a set of some recent contributions to energy equality for the Navier-Stokes equations

In these notes we want, in addition to presenting some new results, to both clean up and refine some reflections on a couple of articles published on paper a few years ago, 2019-20. These papers concerned integral sufficient conditions on $u\,,$ $\n u\,,$ and mixed, to guarantee the equality of the energy, EE in the sequel, for solutions of the Navier-Stokes equations under the classical non-slip boundary condition. Concerning the $\n u$ case a crucial role was enjoyed by a previous well known Berselli and Chiodaroli's pioneering 2019 work on the subject. The above three papers are the main sources of these notes. References will be mostly concentrated on their direct relation to the above papers at the time of pubblication. More recent results will be not stated throughout the article. However, in the last section, the reader will be suitably sent to the more recent bibliography. Below, we also turn back to the innovative interpretation of some main parameters which allowed to overcome their apparent incongruence. Non-Newtonian fluids were also considered in our 2019 paper, maybe for the first time in the above Berselli-Chiodaroli's particular $\n u\,$ context. However we will stick mostly to the Newtonian case since in the end we come to the conclusion that there are not particular additional obstacles to extend the present results from Newtonian to non-Newtonian fluids. Hence we avoid to go further in this direction.

math.AP↗
arXiv2023

Biocompatible wearable touch panel based on ionically conductive organic hydrogels with anti-freezing, anti-dehydration, self-healing, and underwater adhesion properties

Next-generation touch panels are poised to benefit from the use of stretchable and transparent soft ionic conductors, but these materials face several challenges in practical application, including structural damage, loss of functionality, and device stratification, particularly in extreme environments. To address these challenges, in this work, a biocompatible, transparent, self-adhesive gelatin-PAA-based organic hydrogel (PC-OH) was developed, the gel can adhere to the skin in both air and underwater conditions and also anti-freezing, anti-drying, fast self-healable (with a self-healing time of less than 4s in air and underwater), long-term stable for up to 7 days at a wide range of temperatures, highly stretchable, and conductive over a wide temperature range. Using this organic hydrogel, an organic hydrogel-based surface capacitive touch system has been developed that can detect finger touch position in wet environments and over a wide temperature range, demonstrating its ability to sense finger touch position. The wearable touch panel has been successfully demonstrated through the ability to write text, draw figures, and play electronic games, showcasing its potential use in various applications. This breakthrough has significant implications for the development of next-generation touch panels, particularly in the healthcare, sports, and entertainment industries, where reliable and versatile human-machine interfaces are essential.

cond-mat.soft↗
arXiv2023

MixCycle: Mixup Assisted Semi-Supervised 3D Single Object Tracking with Cycle Consistency

3D single object tracking (SOT) is an indispensable part of automated driving. Existing approaches rely heavily on large, densely labeled datasets. However, annotating point clouds is both costly and time-consuming. Inspired by the great success of cycle tracking in unsupervised 2D SOT, we introduce the first semi-supervised approach to 3D SOT. Specifically, we introduce two cycle-consistency strategies for supervision: 1) Self tracking cycles, which leverage labels to help the model converge better in the early stages of training; 2) forward-backward cycles, which strengthen the tracker's robustness to motion variations and the template noise caused by the template update strategy. Furthermore, we propose a data augmentation strategy named SOTMixup to improve the tracker's robustness to point cloud diversity. SOTMixup generates training samples by sampling points in two point clouds with a mixing rate and assigns a reasonable loss weight for training according to the mixing rate. The resulting MixCycle approach generalizes to appearance matching-based trackers. On the KITTI benchmark, based on the P2B tracker, MixCycle trained with $\textbf{10\%}$ labels outperforms P2B trained with $\textbf{100\%}$ labels, and achieves a $\textbf{28.4\%}$ precision improvement when using $\textbf{1\%}$ labels. Our code will be released at \url{https://github.com/Mumuqiao/MixCycle}.

cs.CV↗
arXiv2023

Revisiting Event-based Video Frame Interpolation

Dynamic vision sensors or event cameras provide rich complementary information for video frame interpolation. Existing state-of-the-art methods follow the paradigm of combining both synthesis-based and warping networks. However, few of those methods fully respect the intrinsic characteristics of events streams. Given that event cameras only encode intensity changes and polarity rather than color intensities, estimating optical flow from events is arguably more difficult than from RGB information. We therefore propose to incorporate RGB information in an event-guided optical flow refinement strategy. Moreover, in light of the quasi-continuous nature of the time signals provided by event cameras, we propose a divide-and-conquer strategy in which event-based intermediate frame synthesis happens incrementally in multiple simplified stages rather than in a single, long stage. Extensive experiments on both synthetic and real-world datasets show that these modifications lead to more reliable and realistic intermediate frame results than previous video frame interpolation methods. Our findings underline that a careful consideration of event characteristics such as high temporal density and elevated noise benefits interpolation accuracy.

cs.CV↗
arXiv2023

3D Registration with Maximal Cliques

As a fundamental problem in computer vision, 3D point cloud registration (PCR) aims to seek the optimal pose to align a point cloud pair. In this paper, we present a 3D registration method with maximal cliques (MAC). The key insight is to loosen the previous maximum clique constraint, and mine more local consensus information in a graph for accurate pose hypotheses generation: 1) A compatibility graph is constructed to render the affinity relationship between initial correspondences. 2) We search for maximal cliques in the graph, each of which represents a consensus set. We perform node-guided clique selection then, where each node corresponds to the maximal clique with the greatest graph weight. 3) Transformation hypotheses are computed for the selected cliques by the SVD algorithm and the best hypothesis is used to perform registration. Extensive experiments on U3M, 3DMatch, 3DLoMatch and KITTI demonstrate that MAC effectively increases registration accuracy, outperforms various state-of-the-art methods and boosts the performance of deep-learned methods. MAC combined with deep-learned methods achieves state-of-the-art registration recall of 95.7% / 78.9% on 3DMatch / 3DLoMatch.

cs.CV↗
arXiv2023

Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes

We present a efficient multi-view inverse rendering method for large-scale real-world indoor scenes that reconstructs global illumination and physically-reasonable SVBRDFs. Unlike previous representations, where the global illumination of large scenes is simplified as multiple environment maps, we propose a compact representation called Texture-based Lighting (TBL). It consists of 3D mesh and HDR textures, and efficiently models direct and infinite-bounce indirect lighting of the entire large scene. Based on TBL, we further propose a hybrid lighting representation with precomputed irradiance, which significantly improves the efficiency and alleviates the rendering noise in the material optimization. To physically disentangle the ambiguity between materials, we propose a three-stage material optimization strategy based on the priors of semantic segmentation and room segmentation. Extensive experiments show that the proposed method outperforms the state-of-the-art quantitatively and qualitatively, and enables physically-reasonable mixed-reality applications such as material editing, editable novel view synthesis and relighting. The project page is at https://lzleejean.github.io/TexIR.

cs.CV↗
arXiv2023

Hierarchical Prior Mining for Non-local Multi-View Stereo

As a fundamental problem in computer vision, multi-view stereo (MVS) aims at recovering the 3D geometry of a target from a set of 2D images. Recent advances in MVS have shown that it is important to perceive non-local structured information for recovering geometry in low-textured areas. In this work, we propose a Hierarchical Prior Mining for Non-local Multi-View Stereo (HPM-MVS). The key characteristics are the following techniques that exploit non-local information to assist MVS: 1) A Non-local Extensible Sampling Pattern (NESP), which is able to adaptively change the size of sampled areas without becoming snared in locally optimal solutions. 2) A new approach to leverage non-local reliable points and construct a planar prior model based on K-Nearest Neighbor (KNN), to obtain potential hypotheses for the regions where prior construction is challenging. 3) A Hierarchical Prior Mining (HPM) framework, which is used to mine extensive non-local prior information at different scales to assist 3D model recovery, this strategy can achieve a considerable balance between the reconstruction of details and low-textured areas. Experimental results on the ETH3D and Tanks \& Temples have verified the superior performance and strong generalization capability of our method. Our code will be released.

cs.CV↗
arXiv2023

PSNet: a deep learning model based digital phase shifting algorithm from a single fringe image

As the gold standard for phase retrieval, phase-shifting algorithm (PS) has been widely used in optical interferometry, fringe projection profilometry, etc. However, capturing multiple fringe patterns in PS limits the algorithm to only a narrow range of application. To this end, a deep learning (DL) model based digital PS algorithm from only a single fringe image is proposed. By training on a simulated dataset of PS fringe patterns, the learnt model, denoted PSNet, can predict fringe patterns with other PS steps when given a pattern with the first PS step. Simulation and experiment results demonstrate the PSNet's promising performance on accurate prediction of digital PS patterns, and robustness to complex scenarios such as surfaces with varying curvature and reflectance.

physics.optics↗
arXiv2023

A High-Throughput Computational Dataset of Halide Perovskite Alloys

Novel halide perovskites with improved stability and optoelectronic properties can be designed via composition engineering at cation and/or anion sites. Data-driven methods, especially high-throughput first principles computations and subsequent analysis based on unique materials descriptors, are key to achieving this goal. In this work, we report a density functional theory (DFT) based dataset of 495 $ABX_3$ halide perovskite compounds, with various atomic and molecular species considered at A, B and X sites, and different amounts of mixing applied at each site using the special quasirandom structures (SQS) approach for alloys. We perform GGA-PBE calculations on all 495 pseudo-cubic perovskite structures and around 250 calculations using the HSE06 functional, with and without spin-orbit coupling, both including geometry optimization and static calculations on PBE optimized structures. Lattice constants, decomposition energy, band gap, and theoretical photovoltaic efficiency, are computed using each level of theory, and comparisons are made with collected experimental values. Trends in the data are unraveled in terms of the effects of mixing at different sites, fractions of specific elemental or molecular species present in the compound, and averaged physical properties of species at different sites. We perform screening across the perovskite dataset based on multiple definitions of tolerance factors, deviation from cubicity, computed stability and optoelectronic properties, leading to a list of promising compositions and design principles for achieving multiple desired properties. Our multi-objective, multi-fidelity, computational halide perovskite alloy dataset, one of the most comprehensive to date, is available open-source, and currently being used to train predictive and optimization models for accelerating the design of novel compositions for superior optoelectronic applications.

cond-mat.mtrl-sci↗
arXiv2023

Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased

There is a recent trend of applying multi-agent reinforcement learning (MARL) to train an agent that can cooperate with humans in a zero-shot fashion without using any human data. The typical workflow is to first repeatedly run self-play (SP) to build a policy pool and then train the final adaptive policy against this pool. A crucial limitation of this framework is that every policy in the pool is optimized w.r.t. the environment reward function, which implicitly assumes that the testing partners of the adaptive policy will be precisely optimizing the same reward function as well. However, human objectives are often substantially biased according to their own preferences, which can differ greatly from the environment reward. We propose a more general framework, Hidden-Utility Self-Play (HSP), which explicitly models human biases as hidden reward functions in the self-play objective. By approximating the reward space as linear functions, HSP adopts an effective technique to generate an augmented policy pool with biased policies. We evaluate HSP on the Overcooked benchmark. Empirical results show that our HSP method produces higher rewards than baselines when cooperating with learned human models, manually scripted policies, and real humans. The HSP policy is also rated as the most assistive policy based on human feedback.

cs.AI↗