arXiv subjects

Hao Ouyang

Publications and source records attributed to Hao Ouyang.

At least 55 records · Page 3Linked to original sources

arXiv2024

InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

3D Gaussians have recently emerged as an efficient representation for novel view synthesis. This work studies its editability with a particular focus on the inpainting task, which aims to supplement an incomplete set of 3D Gaussians with additional points for visually harmonious rendering. Compared to 2D inpainting, the crux of inpainting 3D Gaussians is to figure out the rendering-relevant properties of the introduced points, whose optimization largely benefits from their initial 3D positions. To this end, we propose to guide the point initialization with an image-conditioned depth completion model, which learns to directly restore the depth map based on the observed image. Such a design allows our model to fill in depth values at an aligned scale with the original depth, and also to harness strong generalizability from largescale diffusion prior. Thanks to the more accurate depth completion, our approach, dubbed InFusion, surpasses existing alternatives with sufficiently better fidelity and efficiency under various complex scenarios. We further demonstrate the effectiveness of InFusion with several practical applications, such as inpainting with user-specific texture or with novel object insertion.

cs.CV↗

arXiv2023

Text2Immersion: Generative Immersive Scene with 3D Gaussians

We introduce Text2Immersion, an elegant method for producing high-quality 3D immersive scenes from text prompts. Our proposed pipeline initiates by progressively generating a Gaussian cloud using pre-trained 2D diffusion and depth estimation models. This is followed by a refining stage on the Gaussian cloud, interpolating and refining it to enhance the details of the generated scene. Distinct from prevalent methods that focus on single object or indoor scenes, or employ zoom-out trajectories, our approach generates diverse scenes with various objects, even extending to the creation of imaginary scenes. Consequently, Text2Immersion can have wide-ranging implications for various applications such as virtual reality, game development, and automated content creation. Extensive evaluations demonstrate that our system surpasses other methods in rendering quality and diversity, further progressing towards text-driven 3D scene generation. We will make the source code publicly accessible at the project page.

cs.CV↗

arXiv2023

Fermionic Bogomolnyi-Prasad-Sommerfield Wilson loops in four-dimensional $\mathcal{N}=2$ superconformal gauge theories

We construct for the first time Drukker-Trancanelli (DT) type fermionic Bogomolnyi-Prasad-Sommerfield (BPS) Wilson loops in four-dimensional $\mathcal{N}=2$ superconformal $SU(N)\times SU(N)$ quiver theory and $\mathcal{N}=4$ super Yang-Mills theory. The connections of these fermionic BPS Wilson loops have a supermatrix structure. We construct timelike BPS Wilson lines in Minkowski spacetime and circular BPS Wilson loops in Euclidean space. These Wilson loops involve dimensionful parameters. For generic values of parameters, they preserve one real (complex) supercharge in Lorentzian (Euclidean) signature. Supersymmetry enhancement for Wilson loops happens when the parameters satisfy certain constraints. The nature of such loops is quite different from the Wilson loop operators involving fermions constructed previously in the literature on four-dimensional gauge theories. We hope that further investigations of such new Wilson loops will explore deep structures in both the gauge theories and gauge/gravity dualities.

hep-th↗

arXiv2022

High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization

We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views while preserving specific details of the input image. High-fidelity 3D GAN inversion is inherently challenging due to the geometry-texture trade-off in 3D inversion, where overfitting to a single view input image often damages the estimated geometry during the latent optimization. To solve this challenge, we propose a novel pipeline that builds on the pseudo-multi-view estimation with visibility analysis. We keep the original textures for the visible parts and utilize generative priors for the occluded parts. Extensive experiments show that our approach achieves advantageous reconstruction and novel view synthesis quality over state-of-the-art methods, even for images with out-of-distribution textures. The proposed pipeline also enables image attribute editing with the inverted latent code and 3D-aware texture modification. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.

cs.CV↗

arXiv2022

Pretraining is All You Need for Image-to-Image Translation

We propose to use pretraining to boost general image-to-image translation. Prior image-to-image translation methods usually need dedicated architectural design and train individual translation models from scratch, struggling for high-quality generation of complex scenes, especially when paired training data are not abundant. In this paper, we regard each image-to-image translation problem as a downstream task and introduce a simple and generic framework that adapts a pretrained diffusion model to accommodate various kinds of image-to-image translation. We also propose adversarial training to enhance the texture synthesis in the diffusion model training, in conjunction with normalized guidance sampling to improve the generation quality. We present extensive empirical comparison across various tasks on challenging benchmarks such as ADE20K, COCO-Stuff, and DIODE, showing the proposed pretraining-based image-to-image translation (PITI) is capable of synthesizing images of unprecedented realism and faithfulness.

cs.CV↗

arXiv2022

Note on $T\bar{T}$ deformed matrix models and JT supergravity duals

In this work we calculate the partition functions of $\mathcal{N}=1$ type 0A and 0B JT supergravity (SJT) on 2D surfaces of arbitrary genus with multiple finite cut-off boundaries, based on the $T\bar{T}$ deformed super-Schwarzian theories. In terms of SJT/matrix model duality, we compute the corresponding correlation functions in the $T\bar{T}$ deformed matrix model side by using topological recursion relations as well as the transformation properties of topological recursion relations under $T\bar{T}$ deformation. We check that the partition functions finite cut-off 0A and 0B SJT on generic 2D surfaces match the associated correlation functions in $T\bar{T}$ deformed matrix models respectively.

hep-th↗

arXiv2022

Real-Time Neural Character Rendering with Pose-Guided Multiplane Images

We propose pose-guided multiplane image (MPI) synthesis which can render an animatable character in real scenes with photorealistic quality. We use a portable camera rig to capture the multi-view images along with the driving signal for the moving subject. Our method generalizes the image-to-image translation paradigm, which translates the human pose to a 3D scene representation -- MPIs that can be rendered in free viewpoints, using the multi-views captures as supervision. To fully cultivate the potential of MPI, we propose depth-adaptive MPI which can be learned using variable exposure images while being robust to inaccurate camera registration. Our method demonstrates advantageous novel-view synthesis quality over the state-of-the-art approaches for characters with challenging motions. Moreover, the proposed method is generalizable to novel combinations of training poses and can be explicitly controlled. Our method achieves such expressive and animatable character rendering all in real time, serving as a promising solution for practical applications.

cs.CV↗

arXiv2022

TBA-like equations for non-planar scattering amplitude/Wilson lines duality at strong coupling

We compute the minimal area of a string worldsheet ending on two infinite periodic light-like Wilson lines in the AdS$_3$ boundary, which is dual to the first non-planar correction to the gluon scattering amplitude in $\mathcal{N}=4$ SYM at the strong coupling. Using the connection between the Hitchin system and the thermodynamic Bethe ansatz (TBA) equations, we present an analytic method to compute the minimal area surface and express the non-trivial part of the minimal area in terms of the free energy of the TBA-like equations. Given the cross ratios as inputs, the area computed from the TBA-like equations matches that calculated using the numerical integration.

hep-th↗

arXiv2022

Deep Video Prior for Video Consistency and Propagation

Applying an image processing algorithm independently to each video frame often leads to temporal inconsistency in the resulting video. To address this issue, we present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly instead of a large dataset. Unlike most previous methods that enforce temporal consistency with optical flow, we show that temporal consistency can be achieved by training a convolutional neural network on a video with Deep Video Prior (DVP). Moreover, a carefully designed iteratively reweighted training strategy is proposed to address the challenging multimodal inconsistency problem. We demonstrate the effectiveness of our approach on 7 computer vision tasks on videos. Extensive quantitative and perceptual experiments show that our approach obtains superior performance than state-of-the-art methods on blind video temporal consistency. We further extend DVP to video propagation and demonstrate its effectiveness in propagating three different types of information (color, artistic style, and object segmentation). A progressive propagation strategy with pseudo labels is also proposed to enhance DVP's performance on video propagation. Our source codes are publicly available at https://github.com/ChenyangLEI/deep-video-prior.

cs.CV↗

arXiv2021

Internal Video Inpainting by Implicit Long-range Propagation

We propose a novel framework for video inpainting by adopting an internal learning strategy. Unlike previous methods that use optical flow for cross-frame context propagation to inpaint unknown regions, we show that this can be achieved implicitly by fitting a convolutional neural network to known regions. Moreover, to handle challenging sequences with ambiguous backgrounds or long-term occlusion, we design two regularization terms to preserve high-frequency details and long-term temporal consistency. Extensive experiments on the DAVIS dataset demonstrate that the proposed method achieves state-of-the-art inpainting quality quantitatively and qualitatively. We further extend the proposed method to another challenging task: learning to remove an object from a video giving a single object mask in only one frame in a 4K video.

cs.CV↗

arXiv2021

Neural Camera Simulators

We present a controllable camera simulator based on deep neural networks to synthesize raw image data under different camera settings, including exposure time, ISO, and aperture. The proposed simulator includes an exposure module that utilizes the principle of modern lens designs for correcting the luminance level. It also contains a noise module using the noise level function and an aperture module with adaptive attention to simulate the side effects on noise and defocus blur. To facilitate the learning of a simulator model, we collect a dataset of the 10,000 raw images of 450 scenes with different exposure settings. Quantitative experiments and qualitative comparisons show that our approach outperforms relevant baselines in raw data synthesize on multiple cameras. Furthermore, the camera simulator enables various applications, including large-aperture enhancement, HDR, auto exposure, and data augmentation for training local feature detectors. Our work represents the first attempt to simulate a camera sensor's behavior leveraging both the advantage of traditional raw sensor features and the power of data-driven deep learning.

cs.CV↗

arXiv2021

Image Inpainting with External-internal Learning and Monochromic Bottleneck

Although recent inpainting approaches have demonstrated significant improvements with deep neural networks, they still suffer from artifacts such as blunt structures and abrupt colors when filling in the missing regions. To address these issues, we propose an external-internal inpainting scheme with a monochromic bottleneck that helps image inpainting models remove these artifacts. In the external learning stage, we reconstruct missing structures and details in the monochromic space to reduce the learning dimension. In the internal learning stage, we propose a novel internal color propagation method with progressive learning strategies for consistent color restoration. Extensive experiments demonstrate that our proposed scheme helps image inpainting models produce more structure-preserved and visually compelling results.

cs.CV↗

arXiv2021

Wilson loops in circular quiver SCFTs at strong coupling

We study circular BPS Wilson loops in the $\mathcal{N}=2$ superconformal $n$-node quiver theories at large $N$ and strong 't Hooft coupling by using localization. We compute the expectation values of Wilson loops in the limit when the 't Hooft couplings are hierarchically different and when they are nearly equal. Based on these results, we make a conjecture for arbitrary strong couplings.

hep-th↗

arXiv2020

$T\bar{T}$ deformation of chiral bosons and Chern-Simons AdS$_3$ gravity

We study the $T\bar{T}$ deformation of the chiral bosons and show the equivalence between the chiral bosons of opposite chiralities and the scalar fields at the Hamiltonian level under the deformation. We also derive the deformed Lagrangian of more generic theories which contain an arbitrary number of chiral bosons to all orders. By using these results, we derive the $T\bar{T}$ deformed boundary action of the AdS$_3$ gravity theory in the Chern-Simons formulation. We compute the deformed one-loop torus partition function, which satisfies the $T\bar{T}$ flow equation up to the one-loop order. Finally, we calculate the deformed stress tensor of a solution describing a BTZ black hole in the boundary theory, which coincides with the boundary stress tensor derived from the BTZ black hole with a finite cutoff.

hep-th↗

arXiv2019

Open Spin Chains from Determinant Like Operators in ABJM Theory

We study the mixing problem of the determinant like operators in ABJM theory to two loop order in the scalar sector. The gravity duals of these operators are open strings attached to the maximal giant graviton, which is a D4-brane wrapping a $\mathbb{CP}^2$ inside $\mathbb{CP}^3$ in our case. The anomalous dimension matrix of these operators can be regarded as an open spin chain Hamiltonian. We provide strong evidence of its integrability based on coordinate Bethe ansatz method and boundary Yang-Baxter equation.

hep-th↗

arXiv2019

Two-Loop Integrability of ABJM Open Spin Chain from Giant Graviton

We prove the integrability of the two-loop open spin chain Hamiltonian from ABJM determinant like operators given in arXiv:1809.09941. By explicitly constructing R-matrices and K-matrices, we successfully obtain the two-loop Hamiltonian from the double row transfer matrices. This proves the integrability of our two-loop Hamiltonian. Based on the vacuum eigenvalues of the transfer matrices, we make a conjecture on the eigenvalues of the transfer matrices for general excited states. Bethe ansatz equations are simply obtained from the analytic conditions at the superficial poles of the eigenvalues.

hep-th↗

arXiv2019

Holographic four-point functions in Toda field theories in $AdS_2$

We consider Toda field theories in a classical Euclidean $AdS_2$ background. We compute the four-point functions of boundary operators in the $a_1$, $a_2$ and $b_2$ Toda field theories. They take the same form as the four-point functions of generators in the corresponding $\mathcal{W}$-algebras. Therefore we conjecture that the boundary operators are in one-to-one correspondence with the generators in the $\mathcal{W}$-algebras.

hep-th↗

arXiv2019

Note on soft theorems and memories in even dimensions

Recently, it has been shown that the Weinberg's formula for soft graviton production is essentially a Fourier transformation of the formula for gravitational memory which provides an effective way to understand how the classical calculation arises as a limiting case of the quantum result. In this note, we propose a general framework that connects the soft theorems to the radiation fields obtained from classical computation for different theories in even dimensions. We show that the latter is nothing but Fourier transformation of the former. The memory formulas can be derived from radiation fields explicitly.

hep-th↗