arXiv subjects

Zhaoxi Chen

Publications and source records attributed to Zhaoxi Chen.

At least 73 records · Page 4Linked to original sources

arXiv2024

3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors

We present a two-stage text-to-3D generation system, namely 3DTopia, which generates high-quality general 3D assets within 5 minutes using hybrid diffusion priors. The first stage samples from a 3D diffusion prior directly learned from 3D data. Specifically, it is powered by a text-conditioned tri-plane latent diffusion model, which quickly generates coarse 3D samples for fast prototyping. The second stage utilizes 2D diffusion priors to further refine the texture of coarse 3D models from the first stage. The refinement consists of both latent and pixel space optimization for high-quality texture generation. To facilitate the training of the proposed system, we clean and caption the largest open-source 3D dataset, Objaverse, by combining the power of vision language models and large language models. Experiment results are reported qualitatively and quantitatively to show the performance of the proposed system. Our codes and models are available at https://github.com/3DTopia/3DTopia

cs.CV↗

arXiv2024

Advancing large-scale thin-film PPLN nonlinear photonics with segmented tunable micro-heaters

Thin-film periodically poled lithium niobate (TF-PPLN) devices have recently gained prominence for efficient wavelength conversion processes in both classical and quantum applications. However, the patterning and poling of TF-PPLN devices today are mostly performed at chip scales, presenting a significant bottleneck for future large-scale nonlinear photonic systems that require the integration of multiple nonlinear components with consistent performance and low cost. Here, we take a pivotal step towards this goal by developing a wafer-scale TF-PPLN nonlinear photonic platform, leveraging ultraviolet stepper lithography and an automated poling process. To address the inhomogeneous broadening of the quasi-phase matching (QPM) spectrum induced by film thickness variations across the wafer, we propose and demonstrate segmented thermal optic tuning modules that can precisely adjust and align the QPM peak wavelengths in each section. \hl{Using the segmented micro-heaters, we show the successful realignment of inhomogeneously broadened multi-peak QPM spectra with up to 57$\%$ enhancement of conversion efficiency. We achieve a high normalized conversion efficiency of 3802$\%$W$^{-1}$cm$^{-2}$ in a 6 mm long PPLN waveguide, recovering 84$\%$ of the theoretically predicted efficiency in this device.} The advanced fabrication techniques and segmented tuning architectures presented herein pave the way for wafer-scale integration of complex functional nonlinear photonic circuits with applications in quantum information processing, precision sensing and metrology, and low-noise-figure optical signal amplification.

physics.optics↗

arXiv2024

LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation

3D content creation has achieved significant progress in terms of both quality and speed. Although current feed-forward models can produce 3D objects in seconds, their resolution is constrained by the intensive computation required during training. In this paper, we introduce Large Multi-View Gaussian Model (LGM), a novel framework designed to generate high-resolution 3D models from text prompts or single-view images. Our key insights are two-fold: 1) 3D Representation: We propose multi-view Gaussian features as an efficient yet powerful representation, which can then be fused together for differentiable rendering. 2) 3D Backbone: We present an asymmetric U-Net as a high-throughput backbone operating on multi-view images, which can be produced from text or single-view image input by leveraging multi-view diffusion models. Extensive experiments demonstrate the high fidelity and efficiency of our approach. Notably, we maintain the fast speed to generate 3D objects within 5 seconds while boosting the training resolution to 512, thereby achieving high-resolution 3D content generation.

cs.CV↗

arXiv2024

URHand: Universal Relightable Hands

Existing photorealistic relightable hand models require extensive identity-specific observations in different views, poses, and illuminations, and face challenges in generalizing to natural illuminations and novel identities. To bridge this gap, we present URHand, the first universal relightable hand model that generalizes across viewpoints, poses, illuminations, and identities. Our model allows few-shot personalization using images captured with a mobile phone, and is ready to be photorealistically rendered under novel illuminations. To simplify the personalization process while retaining photorealism, we build a powerful universal relightable prior based on neural relighting from multi-view images of hands captured in a light stage with hundreds of identities. The key challenge is scaling the cross-identity training while maintaining personalized fidelity and sharp details without compromising generalization under natural illuminations. To this end, we propose a spatially varying linear lighting model as the neural renderer that takes physics-inspired shading as input feature. By removing non-linear activations and bias, our specifically designed lighting model explicitly keeps the linearity of light transport. This enables single-stage training from light-stage data while generalizing to real-time rendering under arbitrary continuous illuminations across diverse identities. In addition, we introduce the joint learning of a physically based model and our neural relighting model, which further improves fidelity and generalization. Extensive experiments show that our approach achieves superior performance over existing methods in terms of both quality and generalizability. We also demonstrate quick personalization of URHand from a short phone scan of an unseen identity.

cs.CV↗

arXiv2024

Integrated lithium niobate microwave photonic processing engine

Integrated microwave photonics is an intriguing field that leverages integrated photonic technologies for the generation, transmission, and manipulation of microwave signals in chip-scale optical systems. In particular, ultrafast processing and computation of analog electronic signals in the optical domain with high fidelity and low latency could enable a variety of applications such as MWP filters, microwave signal processing, and image recognition. An ideal photonic platform for achieving these integrated MWP processing tasks shall simultaneously offer an efficient, linear and high-speed electro-optic modulation block to faithfully perform microwave-optic conversion at low power, and a low-loss functional photonic network that can be configured for a variety of signal processing tasks, as well as large-scale, low-cost manufacturability to monolithically integrate the two building blocks on the same chip. In this work, we demonstrate such an integrated MWP processing engine based on a thin-film lithium niobate platform capable of performing multi-purpose processing and computation tasks of analog signals up to 92 giga samples per second at CMOS-compatible voltages. We demonstrate high-speed analog computation, i.e., first- and second-order temporal integration and differentiation with computing accuracies up to 98.1 %, and deploy these functions to showcase three proof-of-concept applications, namely, ordinary differential equation solving, ultra-wideband signal generation and high-speed edge detection of images. We further leverage the image edge detector to enable a photonic-assisted image segmentation model that could effectively outline the boundaries of melanoma lesion in medical diagnostic images, achieving orders of magnitude faster processing speed and lower power consumption than conventional electronic processors.

physics.optics↗

arXiv2024

Spectral engineering of optical microresonators in anisotropic lithium niobate crystal

On-chip optical microresonators are essential building blocks in integrated optics. The ability to arbitrarily engineer their resonant frequencies is crucial for exploring novel physics in synthetic frequency dimensions and practical applications like nonlinear optical parametric processes and dispersion-engineered frequency comb generation. Photonic crystal ring (PhCR) resonators are a versatile tool for such arbitrary frequency engineering, by controllably creating mode splitting at selected resonances. To date, these PhCRs have mostly been demonstrated in isotropic photonic materials, while such engineering could be significantly more complicated in anisotropic platforms that often offer more fruitful optical properties. Here, we realize the spectral engineering of chip-scale optical microresonators in the anisotropic lithium niobate (LN) crystal by a gradient design that precisely compensates for variations in both refractive index and perturbation strength. We experimentally demonstrate controllable frequency splitting at single and multiple selected resonances in LN PhCR resonators with different sizes, while maintaining high Q-factors up to 1 million. Moreover, we experimentally construct a sharp boundary in the synthetic frequency dimension based on an actively modulated x-cut LN gradient-PhCR, opening up new paths toward the arbitrary control of electro-optic comb spectral shapes and exploration of novel physics in the frequency degree of freedom.

physics.optics↗

arXiv2023

PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generation

We present PrimDiffusion, the first diffusion-based framework for 3D human generation. Devising diffusion models for 3D human generation is difficult due to the intensive computational cost of 3D representations and the articulated topology of 3D humans. To tackle these challenges, our key insight is operating the denoising diffusion process directly on a set of volumetric primitives, which models the human body as a number of small volumes with radiance and kinematic information. This volumetric primitives representation marries the capacity of volumetric representations with the efficiency of primitive-based rendering. Our PrimDiffusion framework has three appealing properties: 1) compact and expressive parameter space for the diffusion model, 2) flexible 3D representation that incorporates human prior, and 3) decoder-free rendering for efficient novel-view and novel-pose synthesis. Extensive experiments validate that PrimDiffusion outperforms state-of-the-art methods in 3D human generation. Notably, compared to GAN-based methods, our PrimDiffusion supports real-time rendering of high-quality 3D humans at a resolution of $512\times512$ once the denoising process is done. We also demonstrate the flexibility of our framework on training-free conditional generation such as texture transfer and 3D inpainting.

cs.CV↗

arXiv2023

SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections

In this work, we present SceneDreamer, an unconditional generative model for unbounded 3D scenes, which synthesizes large-scale 3D landscapes from random noise. Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations. At the core of SceneDreamer is a principled learning paradigm comprising 1) an efficient yet expressive 3D scene representation, 2) a generative scene parameterization, and 3) an effective renderer that can leverage the knowledge from 2D images. Our approach begins with an efficient bird's-eye-view (BEV) representation generated from simplex noise, which includes a height field for surface elevation and a semantic field for detailed scene semantics. This BEV scene representation enables 1) representing a 3D scene with quadratic complexity, 2) disentangled geometry and semantics, and 3) efficient training. Moreover, we propose a novel generative neural hash grid to parameterize the latent space based on 3D positions and scene semantics, aiming to encode generalizable features across various scenes. Lastly, a neural volumetric renderer, learned from 2D image collections through adversarial training, is employed to produce photorealistic images. Extensive experiments demonstrate the effectiveness of SceneDreamer and superiority over state-of-the-art methods in generating vivid yet diverse unbounded 3D worlds.

cs.CV↗

arXiv2023

PERF: Panoramic Neural Radiance Field from a Single Panorama

Neural Radiance Field (NeRF) has achieved substantial progress in novel view synthesis given multi-view images. Recently, some works have attempted to train a NeRF from a single image with 3D priors. They mainly focus on a limited field of view with a few occlusions, which greatly limits their scalability to real-world 360-degree panoramic scenarios with large-size occlusions. In this paper, we present PERF, a 360-degree novel view synthesis framework that trains a panoramic neural radiance field from a single panorama. Notably, PERF allows 3D roaming in a complex scene without expensive and tedious image collection. To achieve this goal, we propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene. Specifically, we first predict a panoramic depth map as initialization given a single panorama and reconstruct visible 3D regions with volume rendering. Then we introduce a collaborative RGBD inpainting approach into a NeRF for completing RGB images and depth maps from random views, which is derived from an RGB Stable Diffusion model and a monocular depth estimator. Finally, we introduce an inpainting-and-erasing strategy to avoid inconsistent geometry between a newly-sampled view and reference views. The two components are integrated into the learning of NeRFs in a unified optimization framework and achieve promising results. Extensive experiments on Replica and a new dataset PERF-in-the-wild demonstrate the superiority of our PERF over state-of-the-art methods. Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications. Project page and code are available at https://perf-project.github.io/ and https://github.com/perf-project/PeRF.

cs.CV↗

arXiv2023

SynBody: Synthetic Dataset with Layered Human Models for 3D Human Perception and Modeling

Synthetic data has emerged as a promising source for 3D human research as it offers low-cost access to large-scale human datasets. To advance the diversity and annotation quality of human models, we introduce a new synthetic dataset, SynBody, with three appealing features: 1) a clothed parametric human model that can generate a diverse range of subjects; 2) the layered human representation that naturally offers high-quality 3D annotations to support multiple tasks; 3) a scalable system for producing realistic data to facilitate real-world tasks. The dataset comprises 1.2M images with corresponding accurate 3D annotations, covering 10,000 human body models, 1,187 actions, and various viewpoints. The dataset includes two subsets for human pose and shape estimation as well as human neural rendering. Extensive experiments on SynBody indicate that it substantially enhances both SMPL and SMPL-X estimation. Furthermore, the incorporation of layered annotations offers a valuable training resource for investigating the Human Neural Radiance Fields (NeRF).

cs.CV↗

arXiv2023

ReliTalk: Relightable Talking Portrait Generation from a Single Video

Recent years have witnessed great progress in creating vivid audio-driven portraits from monocular videos. However, how to seamlessly adapt the created video avatars to other scenarios with different backgrounds and lighting conditions remains unsolved. On the other hand, existing relighting studies mostly rely on dynamically lighted or multi-view data, which are too expensive for creating video portraits. To bridge this gap, we propose ReliTalk, a novel framework for relightable audio-driven talking portrait generation from monocular videos. Our key insight is to decompose the portrait's reflectance from implicitly learned audio-driven facial normals and images. Specifically, we involve 3D facial priors derived from audio features to predict delicate normal maps through implicit functions. These initially predicted normals then take a crucial part in reflectance decomposition by dynamically estimating the lighting condition of the given video. Moreover, the stereoscopic face representation is refined using the identity-consistent loss under simulated multiple lighting conditions, addressing the ill-posed problem caused by limited views available from a single monocular video. Extensive experiments validate the superiority of our proposed framework on both real and synthetic datasets. Our code is released in https://github.com/arthur-qiu/ReliTalk.

cs.CV↗

arXiv2023

SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis

Neural Radiance Field (NeRF) significantly degrades when only a limited number of views are available. To complement the lack of 3D information, depth-based models, such as DSNeRF and MonoSDF, explicitly assume the availability of accurate depth maps of multiple views. They linearly scale the accurate depth maps as supervision to guide the predicted depth of few-shot NeRFs. However, accurate depth maps are difficult and expensive to capture due to wide-range depth distances in the wild. In this work, we present a new Sparse-view NeRF (SparseNeRF) framework that exploits depth priors from real-world inaccurate observations. The inaccurate depth observations are either from pre-trained depth models or coarse depth maps of consumer-level depth sensors. Since coarse depth maps are not strictly scaled to the ground-truth depth maps, we propose a simple yet effective constraint, a local depth ranking method, on NeRFs such that the expected depth ranking of the NeRF is consistent with that of the coarse depth maps in local patches. To preserve the spatial continuity of the estimated depth of NeRF, we further propose a spatial continuity constraint to encourage the consistency of the expected depth continuity of NeRF with coarse depth maps. Surprisingly, with simple depth ranking constraints, SparseNeRF outperforms all state-of-the-art few-shot NeRF methods (including depth-based models) on standard LLFF and DTU datasets. Moreover, we collect a new dataset NVS-RGBD that contains real-world depth maps from Azure Kinect, ZED 2, and iPhone 13 Pro. Extensive experiments on NVS-RGBD dataset also validate the superiority and generalizability of SparseNeRF. Code and dataset are available at https://sparsenerf.github.io/.

cs.CV↗

arXiv2023

Text2Light: Zero-Shot Text-Driven HDR Panorama Generation

High-quality HDRIs(High Dynamic Range Images), typically HDR panoramas, are one of the most popular ways to create photorealistic lighting and 360-degree reflections of 3D scenes in graphics. Given the difficulty of capturing HDRIs, a versatile and controllable generative model is highly desired, where layman users can intuitively control the generation process. However, existing state-of-the-art methods still struggle to synthesize high-quality panoramas for complex scenes. In this work, we propose a zero-shot text-driven framework, Text2Light, to generate 4K+ resolution HDRIs without paired training data. Given a free-form text as the description of the scene, we synthesize the corresponding HDRI with two dedicated steps: 1) text-driven panorama generation in low dynamic range(LDR) and low resolution, and 2) super-resolution inverse tone mapping to scale up the LDR panorama both in resolution and dynamic range. Specifically, to achieve zero-shot text-driven panorama generation, we first build dual codebooks as the discrete representation for diverse environmental textures. Then, driven by the pre-trained CLIP model, a text-conditioned global sampler learns to sample holistic semantics from the global codebook according to the input text. Furthermore, a structure-aware local sampler learns to synthesize LDR panoramas patch-by-patch, guided by holistic semantics. To achieve super-resolution inverse tone mapping, we derive a continuous representation of 360-degree imaging from the LDR panorama as a set of structured latent codes anchored to the sphere. This continuous representation enables a versatile module to upscale the resolution and dynamic range simultaneously. Extensive experiments demonstrate the superior capability of Text2Light in generating high-quality HDR panoramas. In addition, we show the feasibility of our work in realistic rendering and immersive VR.

cs.CV↗

arXiv2023

F$^{2}$-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories

This paper presents a novel grid-based NeRF called F2-NeRF (Fast-Free-NeRF) for novel view synthesis, which enables arbitrary input camera trajectories and only costs a few minutes for training. Existing fast grid-based NeRF training frameworks, like Instant-NGP, Plenoxels, DVGO, or TensoRF, are mainly designed for bounded scenes and rely on space warping to handle unbounded scenes. Existing two widely-used space-warping methods are only designed for the forward-facing trajectory or the 360-degree object-centric trajectory but cannot process arbitrary trajectories. In this paper, we delve deep into the mechanism of space warping to handle unbounded scenes. Based on our analysis, we further propose a novel space-warping method called perspective warping, which allows us to handle arbitrary trajectories in the grid-based NeRF framework. Extensive experiments demonstrate that F2-NeRF is able to use the same perspective warping to render high-quality images on two standard datasets and a new free trajectory dataset collected by us. Project page: https://totoro97.github.io/projects/f2-nerf.

cs.CV↗

arXiv2022

Inverse-designed lithium niobate nanophotonics

Lithium niobate-on-insulator (LNOI) is an emerging photonic platform that exhibits favorable material properties (such as low optical loss, strong nonlinearities, and stability) and enables large-scale integration with stronger optical confinement, showing promise for future optical networks, quantum processors, and nonlinear optical systems. However, while photonics engineering has entered the era of automated "inverse design" via optimization in recent years, the design of LNOI integrated photonic devices still mostly relies on intuitive models and inefficient parameter sweeps, limiting the accessible parameter space, performance, and functionality. Here, we develop and implement a 3D gradient-based inverse-design model tailored for topology optimization of the LNOI platform, which not only could efficiently search a large parameter space but also takes into account practical fabrication constraints, including minimum feature sizes and etched sidewall angles. We experimentally demonstrate a spatial-mode multiplexer, a waveguide crossing, and a compact waveguide bend, all with low insertion losses, tiny footprints, and excellent agreement between simulation and experimental results. The devices, together with the design methodology, represent a crucial step towards the variety of advanced device functionalities needed in future LNOI photonics, and could provide compact and cost-effective solutions for future optical links, quantum technologies and nonlinear optics.

physics.optics↗

arXiv2022

EVA3D: Compositional 3D Human Generation from 2D Image Collections

Inverse graphics aims to recover 3D models from 2D observations. Utilizing differentiable rendering, recent 3D-aware generative models have shown impressive results of rigid object generation using 2D images. However, it remains challenging to generate articulated objects, like human bodies, due to their complexity and diversity in poses and appearances. In this work, we propose, EVA3D, an unconditional 3D human generative model learned from 2D image collections only. EVA3D can sample 3D humans with detailed geometry and render high-quality images (up to 512x256) without bells and whistles (e.g. super resolution). At the core of EVA3D is a compositional human NeRF representation, which divides the human body into local parts. Each part is represented by an individual volume. This compositional representation enables 1) inherent human priors, 2) adaptive allocation of network parameters, 3) efficient training and rendering. Moreover, to accommodate for the characteristics of sparse 2D human image collections (e.g. imbalanced pose distribution), we propose a pose-guided sampling strategy for better GAN learning. Extensive experiments validate that EVA3D achieves state-of-the-art 3D human generation performance regarding both geometry and texture quality. Notably, EVA3D demonstrates great potential and scalability to "inverse-graphics" diverse human bodies with a clean framework.

cs.CV↗

arXiv2022

A power-efficient integrated lithium niobate electro-optic comb generator

Integrated electro-optic (EO) frequency combs are essential components for future applications in optical communications, light detection and ranging, optical computation, sensing and spectroscopy. To date, broadband on-chip EO combs are typically generated in high-quality-factor micro-resonators, while the more straightforward and flexible non-resonant method, usually using single or cascaded EO phase modulators, often requires high driving power to realize a reasonably strong modulation index. Here, we show that the phase modulation efficiency of an integrated lithium niobate modulator could be dramatically enhanced by passing optical signals through the modulation electrodes for a total of 4 round trips, via multiple low-loss TE0/TE1 mode multiplexers and waveguide crossings, reducing electrical power consumption by more than one order of magnitude. Using devices fabricated from a wafer-scale stepper lithography process, we demonstrate a broadband optical frequency comb featuring 47 comb lines at a 25-GHz repetition rate, using a moderate RF driving power of 28 dBm (0.63 W). Leveraging the excellent tunability in repetition rate and operation wavelength, our power-efficient EO comb generator could serve as a compact low-cost solution for future high-speed data transmission, sensing and spectroscopy, as well as classical and quantum optical computation systems.

physics.optics↗

arXiv2022

Relighting4D: Neural Relightable Human from Videos

Human relighting is a highly desirable yet challenging task. Existing works either require expensive one-light-at-a-time (OLAT) captured data using light stage or cannot freely change the viewpoints of the rendered body. In this work, we propose a principled framework, Relighting4D, that enables free-viewpoints relighting from only human videos under unknown illuminations. Our key insight is that the space-time varying geometry and reflectance of the human body can be decomposed as a set of neural fields of normal, occlusion, diffuse, and specular maps. These neural fields are further integrated into reflectance-aware physically based rendering, where each vertex in the neural field absorbs and reflects the light from the environment. The whole framework can be learned from videos in a self-supervised manner, with physically informed priors designed for regularization. Extensive experiments on both real and synthetic datasets demonstrate that our framework is capable of relighting dynamic human actors with free-viewpoints.

cs.CV↗