arXiv subjects

Chen Guo

Publications and source records attributed to Chen Guo.

At least 37 records · Page 2Linked to original sources

arXiv2024

SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation

In this paper, we propose SRIF, a novel Semantic shape Registration framework based on diffusion-based Image morphing and Flow estimation. More concretely, given a pair of extrinsically aligned shapes, we first render them from multi-views, and then utilize an image interpolation framework based on diffusion models to generate sequences of intermediate images between them. The images are later fed into a dynamic 3D Gaussian splatting framework, with which we reconstruct and post-process for intermediate point clouds respecting the image morphing processing. In the end, tailored for the above, we propose a novel registration module to estimate continuous normalizing flow, which deforms source shape consistently towards the target, with intermediate point clouds as weak guidance. Our key insight is to leverage large vision models (LVMs) to associate shapes and therefore obtain much richer semantic information on the relationship between shapes than the ad-hoc feature extraction and alignment. As a consequence, SRIF achieves high-quality dense correspondences on challenging shape pairs, but also delivers smooth, semantically meaningful interpolation in between. Empirical evidence justifies the effectiveness and superiority of our method as well as specific design choices. The code is released at https://github.com/rqhuang88/SRIF.

cs.CV↗

arXiv2024

ReLoo: Reconstructing Humans Dressed in Loose Garments from Monocular Video in the Wild

While previous years have seen great progress in the 3D reconstruction of humans from monocular videos, few of the state-of-the-art methods are able to handle loose garments that exhibit large non-rigid surface deformations during articulation. This limits the application of such methods to humans that are dressed in standard pants or T-shirts. Our method, ReLoo, overcomes this limitation and reconstructs high-quality 3D models of humans dressed in loose garments from monocular in-the-wild videos. To tackle this problem, we first establish a layered neural human representation that decomposes clothed humans into a neural inner body and outer clothing. On top of the layered neural representation, we further introduce a non-hierarchical virtual bone deformation module for the clothing layer that can freely move, which allows the accurate recovery of non-rigidly deforming loose clothing. A global optimization jointly optimizes the shape, appearance, and deformations of the human body and clothing via multi-layer differentiable volume rendering. To evaluate ReLoo, we record subjects with dynamically deforming garments in a multi-view capture studio. This evaluation, both on existing and our novel dataset, demonstrates ReLoo's clear superiority over prior art on both indoor datasets and in-the-wild videos.

cs.CV↗

arXiv2024

MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild

We present MultiPly, a novel framework to reconstruct multiple people in 3D from monocular in-the-wild videos. Reconstructing multiple individuals moving and interacting naturally from monocular in-the-wild videos poses a challenging task. Addressing it necessitates precise pixel-level disentanglement of individuals without any prior knowledge about the subjects. Moreover, it requires recovering intricate and complete 3D human shapes from short video sequences, intensifying the level of difficulty. To tackle these challenges, we first define a layered neural representation for the entire scene, composited by individual human and background models. We learn the layered neural representation from videos via our layer-wise differentiable volume rendering. This learning process is further enhanced by our hybrid instance segmentation approach which combines the self-supervised 3D segmentation and the promptable 2D segmentation module, yielding reliable instance segmentation supervision even under close human interaction. A confidence-guided optimization formulation is introduced to optimize the human poses and shape/appearance alternately. We incorporate effective objectives to refine human poses via photometric information and impose physically plausible constraints on human dynamics, leading to temporally consistent 3D reconstructions with high fidelity. The evaluation of our method shows the superiority over prior art on publicly available datasets and in-the-wild videos.

cs.CV↗

arXiv2024

Highly versatile, two-color setup for high-order harmonic generation using spatial light modulators

We present a novel, interferometric, two-color, high-order harmonic generation setup, based on a turn-key Ytterbium-doped femtosecond laser source and its second harmonic. Each interferometer arm contains a spatial light modulator, with individual capabilities to manipulate the spatial beam profiles and to stabilize the relative delay between the fundamental and the second harmonic. Additionally, separate control of the relative power and focusing geometries of the two color beams is implemented to conveniently perform automatized scans of multiple parameters. A live diagnostics system gives continuous information during ongoing measurements.

physics.optics↗

arXiv2024

4D-DRESS: A 4D Dataset of Real-world Human Clothing with Semantic Annotations

The studies of human clothing for digital avatars have predominantly relied on synthetic datasets. While easy to collect, synthetic data often fall short in realism and fail to capture authentic clothing dynamics. Addressing this gap, we introduce 4D-DRESS, the first real-world 4D dataset advancing human clothing research with its high-quality 4D textured scans and garment meshes. 4D-DRESS captures 64 outfits in 520 human motion sequences, amounting to 78k textured scans. Creating a real-world clothing dataset is challenging, particularly in annotating and segmenting the extensive and complex 4D human scans. To address this, we develop a semi-automatic 4D human parsing pipeline. We efficiently combine a human-in-the-loop process with automation to accurately label 4D scans in diverse garments and body movements. Leveraging precise annotations and high-quality garment meshes, we establish several benchmarks for clothing simulation and reconstruction. 4D-DRESS offers realistic and challenging data that complements synthetic sources, paving the way for advancements in research of lifelike human clothing. Website: https://ait.ethz.ch/4d-dress.

cs.CV↗

arXiv2023

Time-resolved photoemission electron microscopy on a ZnO surface using an extreme ultraviolet attosecond pulse pair

Electrons photoemitted by extreme ultraviolet attosecond pulses derive spatially from the first few atomic surface layers and energetically from the valence band and highest atomic orbitals. As a result, it is possible to probe the emission dynamics from a narrow two-dimensional region in the presence of optical fields as well as obtain elemental specific information. However, combining this with spatially-resolved imaging is a long-standing challenge because of the large inherent spectral width of attosecond pulses as well as the difficulty of making them at high repetition rates. Here we demonstrate an attosecond interferometry experiment on a zinc oxide (ZnO) surface using spatially and energetically resolved photoelectrons. We combine photoemission electron microscopy with near-infrared pump - extreme ultraviolet probe laser spectroscopy and resolve the instantaneous phase of an infrared field with high spatial resolution. Our results show how the core level states with low binding energy of ZnO are well suited to perform spatially resolved attosecond interferometry experiments. We observe a distinct phase shift of the attosecond beat signal across the laser focus which we attribute to wavefront differences between the pump and the probe fields at the surface. Our work demonstrates a clear pathway for attosecond interferometry with high spatial resolution at atomic scale surface regions opening up for a detailed understanding of nanometric light-matter interaction.

physics.optics↗

arXiv2023

Measuring the quantum state of photoelectrons

A photoelectron, emitted due to the absorption of light quanta as described by the photoelectric effect, is often characterized experimentally by a classical quantity, its momentum. However, since the photoelectron is a quantum object, its rigorous characterization requires the reconstruction of the complete quantum state, the photoelectron's density matrix. Here, we use quantum state tomography to fully characterize photoelectrons emitted from helium and argon atoms upon absorption of ultrashort, extreme ultraviolet light pulses. While in helium we measure a pure photoelectronic state, in argon, spin-orbit interaction induces entanglement between the ion and the photoelectron, leading to a reduced purity of the photoelectron state. Our work shows how state tomography gives new insights into the fundamental quantum aspects of light-induced electronic processes in matter, bridging the fields of photoelectron spectroscopy and quantum information, and offering new spectroscopic possibilities for quantum technology.

quant-ph↗

arXiv2023

Measurement of Ultrashort Laser Pulses With a Time-Dependent Polarization State Using the D-Scan Technique

The dispersion scan (d-scan) technique is extended to measurement of the timedependent polarization state of ultrashort laser pulses. In the simplest implementation for linearly polarized ultrashort pulses, the d-scan technique records the second harmonic generation (SHG) spectrum as a function of a known spectral phase manipulation. By applying this method to two orthogonally polarized projections of an arbitrary polarized electric field and by measuring the spectrum at an intermediate angle, we can reconstruct the evolution over time of the polarization state. We demonstrate the method by measuring a polarization gate generated from 6 fs pulses with a combination of waveplates. The measurements are compared to simulations, showing an excellent agreement.

physics.optics↗

arXiv2023

EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild

We present EMDB, the Electromagnetic Database of Global 3D Human Pose and Shape in the Wild. EMDB is a novel dataset that contains high-quality 3D SMPL pose and shape parameters with global body and camera trajectories for in-the-wild videos. We use body-worn, wireless electromagnetic (EM) sensors and a hand-held iPhone to record a total of 58 minutes of motion data, distributed over 81 indoor and outdoor sequences and 10 participants. Together with accurate body poses and shapes, we also provide global camera poses and body root trajectories. To construct EMDB, we propose a multi-stage optimization procedure, which first fits SMPL to the 6-DoF EM measurements and then refines the poses via image observations. To achieve high-quality results, we leverage a neural implicit avatar model to reconstruct detailed human surface geometry and appearance, which allows for improved alignment and smoothness via a dense pixel-level objective. Our evaluations, conducted with a multi-view volumetric capture system, indicate that EMDB has an expected accuracy of 2.3 cm positional and 10.6 degrees angular error, surpassing the accuracy of previous in-the-wild datasets. We evaluate existing state-of-the-art monocular RGB methods for camera-relative and global pose estimation on EMDB. EMDB is publicly available under https://ait.ethz.ch/emdb

cs.CV↗

arXiv2023

Hi4D: 4D Instance Segmentation of Close Human Interaction

We propose Hi4D, a method and dataset for the automatic analysis of physically close human-human interaction under prolonged contact. Robustly disentangling several in-contact subjects is a challenging task due to occlusions and complex shapes. Hence, existing multi-view systems typically fuse 3D surfaces of close subjects into a single, connected mesh. To address this issue we leverage i) individually fitted neural implicit avatars; ii) an alternating optimization scheme that refines pose and surface through periods of close proximity; and iii) thus segment the fused raw scans into individual instances. From these instances we compile Hi4D dataset of 4D textured scans of 20 subject pairs, 100 sequences, and a total of more than 11K frames. Hi4D contains rich interaction-centric annotations in 2D and 3D alongside accurately registered parametric body models. We define varied human pose and shape estimation tasks on this dataset and provide results from state-of-the-art methods on these benchmarks.

cs.CV↗

arXiv2023

X-Avatar: Expressive Human Avatars

We present X-Avatar, a novel avatar model that captures the full expressiveness of digital humans to bring about life-like experiences in telepresence, AR/VR and beyond. Our method models bodies, hands, facial expressions and appearance in a holistic fashion and can be learned from either full 3D scans or RGB-D data. To achieve this, we propose a part-aware learned forward skinning module that can be driven by the parameter space of SMPL-X, allowing for expressive animation of X-Avatars. To efficiently learn the neural shape and deformation fields, we propose novel part-aware sampling and initialization strategies. This leads to higher fidelity results, especially for smaller body parts while maintaining efficient training despite increased number of articulated bones. To capture the appearance of the avatar with high-frequency details, we extend the geometry and deformation fields with a texture network that is conditioned on pose, facial expression, geometry and the normals of the deformed surface. We show experimentally that our method outperforms strong baselines in both data domains both quantitatively and qualitatively on the animation task. To facilitate future research on expressive avatars we contribute a new dataset, called X-Humans, containing 233 sequences of high-quality textured scans from 20 participants, totalling 35,500 data frames.

cs.CV↗

arXiv2023

Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition

We present Vid2Avatar, a method to learn human avatars from monocular in-the-wild videos. Reconstructing humans that move naturally from monocular in-the-wild videos is difficult. Solving it requires accurately separating humans from arbitrary backgrounds. Moreover, it requires reconstructing detailed 3D surface from short video sequences, making it even more challenging. Despite these challenges, our method does not require any groundtruth supervision or priors extracted from large datasets of clothed human scans, nor do we rely on any external segmentation modules. Instead, it solves the tasks of scene decomposition and surface reconstruction directly in 3D by modeling both the human and the background in the scene jointly, parameterized via two separate neural fields. Specifically, we define a temporally consistent human representation in canonical space and formulate a global optimization over the background model, the canonical human shape and texture, and per-frame human pose parameters. A coarse-to-fine sampling strategy for volume rendering and novel objectives are introduced for a clean separation of dynamic human and static background, yielding detailed and robust 3D human geometry reconstructions. We evaluate our methods on publicly available datasets and show improvements over prior art.

cs.CV↗

arXiv2023

Ultra-stable and versatile high-energy resolution setup for attosecond photoelectron spectroscopy

Attosecond photoelectron spectroscopy is often performed with interferometric experimental setups that require outstanding stability. We demonstrate and characterize in detail an actively stabilized, versatile, high spectral resolution attosecond beamline. The active-stabilization system can remain ultra-stable for several hours with an RMS stability of 13 as and a total pump-probe delay scanning range of \sim 400 fs. A tunable femtosecond laser source to drive high-order harmonic generation allows for precisely addressing atomic and molecular resonances. Furthermore, the interferometer includes a spectral shaper in 4f-geometry in the probe arm as well as a tunable bandpass filter in the pump arm, which offer additional high flexibility in terms of tunability as well as narrowband or polychromatic probe pulses. We show that spectral phase measurements of photoelectron wavepackets with the rainbow RABBIT technique (reconstruction of attosecond beating by two photon transitions) with narrowband probe pulses can significantly improve the photoelectron energy resolution. In this setup, the temporal-spectral resolution of photoelectron spectroscopy can reach a new level of accuracy and precision.

physics.atom-ph↗

arXiv2022

Multi-gigawatt peak power post-compression in a bulk multi-pass cell at high repetition rate

The output of a 200 kHz, 34 W, 300 fs Yb amplifier is compressed to 31 fs with > 88 % efficiency to reach a peak power of 2.5 GW, which to date is a record for a single-stage bulk multi-pass cell. Despite operation 80 times above the critical power for self-focusing in bulk material, the setup demonstrates excellent preservation of the input beam quality. Extensive beam and pulse characterizations are performed to show that the compressed pulses are promising drivers for high harmonic generation and nonlinear optics in gases or solids.

physics.optics↗

arXiv2022

PINA: Learning a Personalized Implicit Neural Avatar from a Single RGB-D Video Sequence

We present a novel method to learn Personalized Implicit Neural Avatars (PINA) from a short RGB-D sequence. This allows non-expert users to create a detailed and personalized virtual copy of themselves, which can be animated with realistic clothing deformations. PINA does not require complete scans, nor does it require a prior learned from large datasets of clothed humans. Learning a complete avatar in this setting is challenging, since only few depth observations are available, which are noisy and incomplete (i.e. only partial visibility of the body per frame). We propose a method to learn the shape and non-rigid deformations via a pose-conditioned implicit surface and a deformation field, defined in canonical space. This allows us to fuse all partial observations into a single consistent canonical representation. Fusion is formulated as a global optimization problem over the pose, shape and skinning parameters. The method can learn neural avatars from real noisy RGB-D sequences for a diverse set of people and clothing styles and these avatars can be animated given unseen motion sequences.

cs.CV↗

arXiv2022

Continuous variable quantum state tomography of photoelectrons

We propose a continuous variable quantum state tomography protocol of electrons which result from the ionization of atoms or molecules by the absorption of extreme ultraviolet light pulses. Our protocol is benchmarked against a direct calculation of the quantum state of photoelectrons ejected from helium and argon in the vicinity of a Fano resonance. In the latter case, we furthermore distill ion-photoelectron entanglement due to spin-orbit splitting. This opens new routes towards the investigation of quantum coherence and entanglement properties on the ultrafast timescale.

physics.atom-ph↗

arXiv2021

Human Performance Capture from Monocular Video in the Wild

Capturing the dynamically deforming 3D shape of clothed human is essential for numerous applications, including VR/AR, autonomous driving, and human-computer interaction. Existing methods either require a highly specialized capturing setup, such as expensive multi-view imaging systems, or they lack robustness to challenging body poses. In this work, we propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input. We first build a 3D template human model of the subject based on a learned regression model. We then track this template model's deformation under challenging body articulations based on 2D image observations. Our method outperforms state-of-the-art methods on an in-the-wild human video dataset 3DPW. Moreover, we demonstrate its efficacy in robustness and generalizability on videos from iPER datasets.

cs.CV↗

arXiv2021

Shape-aware Multi-Person Pose Estimation from Multi-View Images

In this paper we contribute a simple yet effective approach for estimating 3D poses of multiple people from multi-view images. Our proposed coarse-to-fine pipeline first aggregates noisy 2D observations from multiple camera views into 3D space and then associates them into individual instances based on a confidence-aware majority voting technique. The final pose estimates are attained from a novel optimization scheme which links high-confidence multi-view 2D observations and 3D joint candidates. Moreover, a statistical parametric body model such as SMPL is leveraged as a regularizing prior for these 3D joint candidates. Specifically, both 3D poses and SMPL parameters are optimized jointly in an alternating fashion. Here the parametric models help in correcting implausible 3D pose estimates and filling in missing joint detections while updated 3D poses in turn guide obtaining better SMPL estimations. By linking 2D and 3D observations, our method is both accurate and generalizes to different data sources because it better decouples the final 3D pose from the inter-person constellation and is more robust to noisy 2D detections. We systematically evaluate our method on public datasets and achieve state-of-the-art performance. The code and video will be available on the project page: https://ait.ethz.ch/projects/2021/multi-human-pose/.

cs.CV↗