arXiv subjects

Jiahui Huang

Publications and source records attributed to Jiahui Huang.

At least 55 records · Page 3Linked to original sources

arXiv2025

GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control

We present GEN3C, a generative video model with precise Camera Control and temporal 3D Consistency. Prior video models already generate realistic videos, but they tend to leverage little 3D information, leading to inconsistencies, such as objects popping in and out of existence. Camera control, if implemented at all, is imprecise, because camera parameters are mere inputs to the neural network which must then infer how the video depends on the camera. In contrast, GEN3C is guided by a 3D cache: point clouds obtained by predicting the pixel-wise depth of seed images or previously generated frames. When generating the next frames, GEN3C is conditioned on the 2D renderings of the 3D cache with the new camera trajectory provided by the user. Crucially, this means that GEN3C neither has to remember what it previously generated nor does it have to infer the image structure from the camera pose. The model, instead, can focus all its generative power on previously unobserved regions, as well as advancing the scene state to the next frame. Our results demonstrate more precise camera control than prior work, as well as state-of-the-art results in sparse-view novel view synthesis, even in challenging settings such as driving scenes and monocular dynamic video. Results are best viewed in videos. Check out our webpage! https://research.nvidia.com/labs/toronto-ai/GEN3C/

cs.CV↗

arXiv2025

MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation

This paper presents a method that allows users to design cinematic video shots in the context of image-to-video generation. Shot design, a critical aspect of filmmaking, involves meticulously planning both camera movements and object motions in a scene. However, enabling intuitive shot design in modern image-to-video generation systems presents two main challenges: first, effectively capturing user intentions on the motion design, where both camera movements and scene-space object motions must be specified jointly; and second, representing motion information that can be effectively utilized by a video diffusion model to synthesize the image animations. To address these challenges, we introduce MotionCanvas, a method that integrates user-driven controls into image-to-video (I2V) generation models, allowing users to control both object and camera motions in a scene-aware manner. By connecting insights from classical computer graphics and contemporary video generation techniques, we demonstrate the ability to achieve 3D-aware motion control in I2V synthesis without requiring costly 3D-related training data. MotionCanvas enables users to intuitively depict scene-space motion intentions, and translates them into spatiotemporal motion-conditioning signals for video diffusion models. We demonstrate the effectiveness of our method on a wide range of real-world image content and shot-design scenarios, highlighting its potential to enhance the creative workflows in digital content creation and adapt to various image and video editing applications.

cs.CV↗

arXiv2024

STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes

We present STORM, a spatio-temporal reconstruction model designed for reconstructing dynamic outdoor scenes from sparse observations. Existing dynamic reconstruction methods often rely on per-scene optimization, dense observations across space and time, and strong motion supervision, resulting in lengthy optimization times, limited generalization to novel views or scenes, and degenerated quality caused by noisy pseudo-labels for dynamics. To address these challenges, STORM leverages a data-driven Transformer architecture that directly infers dynamic 3D scene representations--parameterized by 3D Gaussians and their velocities--in a single forward pass. Our key design is to aggregate 3D Gaussians from all frames using self-supervised scene flows, transforming them to the target timestep to enable complete (i.e., "amodal") reconstructions from arbitrary viewpoints at any moment in time. As an emergent property, STORM automatically captures dynamic instances and generates high-quality masks using only reconstruction losses. Extensive experiments on public datasets show that STORM achieves precise dynamic scene reconstruction, surpassing state-of-the-art per-scene optimization methods (+4.3 to 6.6 PSNR) and existing feed-forward approaches (+2.1 to 4.7 PSNR) in dynamic regions. STORM reconstructs large-scale outdoor scenes in 200ms, supports real-time rendering, and outperforms competitors in scene flow estimation, improving 3D EPE by 0.422m and Acc5 by 28.02%. Beyond reconstruction, we showcase four additional applications of our model, illustrating the potential of self-supervised learning for broader dynamic scene understanding.

cs.CV↗

arXiv2024

Photophysics of O-band and transition metal color centers in monolithic silicon for quantum communications

Color centers in the O-band (1260-1360 nm) are critical for realizing long-coherence quantum network nodes in memory-assisted quantum communications. However, only a limited number of O-band color centers have been explored in silicon hosts as spin-photon interfaces. This study explores and compares two promising O-band defects in silicon: T centers and $^*$Cu (transition metal) color centers. During T center formation, we observed the formation and dissolution of various defects, including the copper-silver-related defect with a doublet line around 1312 nm ($^*$Cu$^{0}_{n}$), near the optical fiber zero dispersion wavelength. We then investigate the photophysics of both T and $^*$Cu centers, focusing on their emission spectra and spin properties to assess their potential for high-fidelity spin-photon interfaces. Additionally, we report a 25\% broadening of the $^*$Cu$^{0}_{0}$ line under a 0.5 T magnetic field, potentially linked to spin degeneracy, suggesting that this defect may provide a promising alternative to T centers for spin-photon interfaces.

quant-ph↗

arXiv2024

SCube: Instant Large-Scale Scene Reconstruction using VoxSplats

We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images. Our method encodes reconstructed scenes using a novel representation VoxSplat, which is a set of 3D Gaussians supported on a high-resolution sparse-voxel scaffold. To reconstruct a VoxSplat from images, we employ a hierarchical voxel latent diffusion model conditioned on the input images followed by a feedforward appearance prediction model. The diffusion model generates high-resolution grids progressively in a coarse-to-fine manner, and the appearance network predicts a set of Gaussians within each voxel. From as few as 3 non-overlapping input images, SCube can generate millions of Gaussians with a 1024^3 voxel grid spanning hundreds of meters in 20 seconds. Past works tackling scene reconstruction from images either rely on per-scene optimization and fail to reconstruct the scene away from input views (thus requiring dense view coverage as input) or leverage geometric priors based on low-resolution models, which produce blurry results. In contrast, SCube leverages high-resolution sparse networks and produces sharp outputs from few views. We show the superiority of SCube compared to prior art using the Waymo self-driving dataset on 3D reconstruction and demonstrate its applications, such as LiDAR simulation and text-to-scene generation.

cs.CV↗

arXiv2024

An X-ray Shell Reveals the Supernova Explosion for Galactic Microquasar SS 433

How black holes are formed remains an open and fundamental question in Astrophysics. Despite theoretical predictions, it lacks observations to understand whether the black hole formation experiences a supernova explosion. Here we report the discovery of an X-ray shell north of the Galactic micro-quasar SS 433 harboring a stellar-mass black hole spatially associated with radio continuum and polarization emissions, and an HI cloud. Its spectrum can be reproduced by a 1-keV under-ionized plasma, from which the shell is inferred to have been created by a supernova explosion 20-30 kyr ago and its properties constitute evidence for canonical SN explosions to create some black holes. Our analysis precludes other possible origins including heated by jets or blown by disk winds. According to the lower mass limit of the compact object in SS 433, we roughly deduced that the progenitor should be more massive than 25 M$_{\odot}$. The existence of such a young remnant in SS 433 can also lead to new insights into the supercritical accretion in young microquasars and the $γ$-ray emission of this system. The fallback ejecta may provide accretion materials within tens of thousands of years while the shock of the supernova remnant may play a crucial role in the cosmic ray (re)acceleration.

astro-ph.HE↗

arXiv2024

Local All-Pair Correspondence for Point Tracking

We introduce LocoTrack, a highly accurate and efficient model designed for the task of tracking any point (TAP) across video sequences. Previous approaches in this task often rely on local 2D correlation maps to establish correspondences from a point in the query image to a local region in the target image, which often struggle with homogeneous regions or repetitive features, leading to matching ambiguities. LocoTrack overcomes this challenge with a novel approach that utilizes all-pair correspondences across regions, i.e., local 4D correlation, to establish precise correspondences, with bidirectional correspondence and matching smoothness significantly enhancing robustness against ambiguities. We also incorporate a lightweight correlation encoder to enhance computational efficiency, and a compact Transformer architecture to integrate long-term temporal information. LocoTrack achieves unmatched accuracy on all TAP-Vid benchmarks and operates at a speed almost 6 times faster than the current state-of-the-art.

cs.CV↗

arXiv2024

XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies

We present XCube (abbreviated as $\mathcal{X}^3$), a novel generative model for high-resolution sparse 3D voxel grids with arbitrary attributes. Our model can generate millions of voxels with a finest effective resolution of up to $1024^3$ in a feed-forward fashion without time-consuming test-time optimization. To achieve this, we employ a hierarchical voxel latent diffusion model which generates progressively higher resolution grids in a coarse-to-fine manner using a custom framework built on the highly efficient VDB data structure. Apart from generating high-resolution objects, we demonstrate the effectiveness of XCube on large outdoor scenes at scales of 100m$\times$100m with a voxel size as small as 10cm. We observe clear qualitative and quantitative improvements over past approaches. In addition to unconditional generation, we show that our model can be used to solve a variety of tasks such as user-guided editing, scene completion from a single scan, and text-to-3D. The source code and more results can be found at https://research.nvidia.com/labs/toronto-ai/xcube/.

cs.CV↗

arXiv2024

Approximately Piecewise E(3) Equivariant Point Networks

Integrating a notion of symmetry into point cloud neural networks is a provably effective way to improve their generalization capability. Of particular interest are $E(3)$ equivariant point cloud networks where Euclidean transformations applied to the inputs are preserved in the outputs. Recent efforts aim to extend networks that are $E(3)$ equivariant, to accommodate inputs made of multiple parts, each of which exhibits local $E(3)$ symmetry. In practical settings, however, the partitioning into individually transforming regions is unknown a priori. Errors in the partition prediction would unavoidably map to errors in respecting the true input symmetry. Past works have proposed different ways to predict the partition, which may exhibit uncontrolled errors in their ability to maintain equivariance to the actual partition. To this end, we introduce APEN: a general framework for constructing approximate piecewise-$E(3)$ equivariant point networks. Our primary insight is that functions that are equivariant with respect to a finer partition will also maintain equivariance in relation to the true partition. Leveraging this observation, we propose a design where the equivariance approximation error at each layers can be bounded solely in terms of (i) uncertainty quantification of the partition prediction, and (ii) bounds on the probability of failing to suggest a proper subpartition of the ground truth one. We demonstrate the effectiveness of APEN using two data types exemplifying part-based symmetry: (i) real-world scans of room scenes containing multiple furniture-type objects; and, (ii) human motions, characterized by articulated parts exhibiting rigid movement. Our empirical results demonstrate the advantage of integrating piecewise $E(3)$ symmetry into network design, showing a distinct improvement in generalization compared to prior works for both classification and segmentation tasks.

cs.LG↗

arXiv2023

High-coherence parallelization in integrated photonics

Coherent optics has profoundly impacted diverse applications ranging from communications, LiDAR to quantum computations. However, building coherent systems in integrated photonics previously came at great expense in hardware integration and energy efficiency: the lack of a power-efficient way to generate highly coherent light necessitates bulky lasers and amplifiers, while frequency and phase recovery schemes require huge digital signal processing resources. In this work, we demonstrate a high-coherence parallelization strategy that facilitates advanced integrated coherent systems at a minimum price. Using a self-injection locked microcomb to injection lock a distributed feedback laser array, we boost the microcomb power by a record high gain of up to 60 dB on chip with no degradation in coherence. This strategy enables tens of highly coherent channels with an intrinsic linewidth down to the 10 Hz level and power of more than 20 dBm. The overall electrical to optical wall-plug efficiency reaches 19%, comparable with that of the state-of-the-art semiconductor lasers. Driven by this parallel source, we demonstrate a silicon photonic communication link with an unprecedented data rate beyond 60 Tbit/s. Importantly, the high coherence we achieve reduces the coherent-related DSP consumption by 99.999% compared with the traditional III-V laser pump scheme. This work paves a way to realizing scalable, high-performance coherent integrated photonic systems, potentially benefiting numerous applications.

physics.optics↗

arXiv2023

Single site-controlled inverted pyramidal InGaAs QD-nanocavity operating at the onset of the strong coupling regime

Precise positioning of single site-controlled inverted pyramidal InGaAs QD at the antinode of a GaAs photonic crystal cavity with nanometer-scale accuracy holds unique advantages compared to self-assembled QDs and offers great promise for practical on-chip photonic quantum information processing. However, the strong coupling regime in this geometry has not yet been achieved due to the low cavity Q-factor based on the (111)B-oriented membrane structures. Here, we reveal the onset of phonon-mediated coherent exciton-photon interaction on our tailored single site-controlled InGaAs QD - photonic crystal cavity. Our results present a Rabi-like oscillation of luminescence intensity between excitonic and photonic components correlated with their energy splitting pronounced at small detuning. Such Rabi-like oscillation is well reproduced by modeling the coherent exchange of the exciton-photon population. The modeling further reveals an oscillatory two-time covariance at QD-cavity resonance, which indicates that the system operates at the onset of the strong coupling regime. Moreover, by using the cavity mode as a probe of the virtual state of the QD induced by phonon scattering, it reveals an increase in phonon scattering rates near the QD-cavity resonance and asymmetric phonon emission and absorption rate even around 50 K.

cond-mat.mes-hall↗

arXiv2023

Revisiting Acceptability Judgements

In this work, we revisit linguistic acceptability in the context of large language models. We introduce CoLAC - Corpus of Linguistic Acceptability in Chinese, the first large-scale acceptability dataset for a non-Indo-European language. It is verified by native speakers and is the first acceptability dataset that comes with two sets of labels: a linguist label and a crowd label. Our experiments show that even the largest InstructGPT model performs only at chance level on CoLAC, while ChatGPT's performance (48.30 MCC) is also much below supervised models (59.03 MCC) and human (65.11 MCC). Through cross-lingual transfer experiments and fine-grained linguistic analysis, we provide detailed analysis of the model predictions and demonstrate for the first time that knowledge of linguistic acceptability can be transferred across typologically distinct languages, as well as be traced back to pre-training. Our dataset is publicly available at \url{https://github.com/huhailinguist/CoLAC}.

cs.CL↗

arXiv2023

DiffFacto: Controllable Part-Based 3D Point Cloud Generation with Cross Diffusion

While the community of 3D point cloud generation has witnessed a big growth in recent years, there still lacks an effective way to enable intuitive user control in the generation process, hence limiting the general utility of such methods. Since an intuitive way of decomposing a shape is through its parts, we propose to tackle the task of controllable part-based point cloud generation. We introduce DiffFacto, a novel probabilistic generative model that learns the distribution of shapes with part-level control. We propose a factorization that models independent part style and part configuration distributions and presents a novel cross-diffusion network that enables us to generate coherent and plausible shapes under our proposed factorization. Experiments show that our method is able to generate novel shapes with multiple axes of control. It achieves state-of-the-art part-level generation quality and generates plausible and coherent shapes while enabling various downstream editing applications such as shape interpolation, mixing, and transformation editing. Project website: https://difffacto.github.io/

cs.CV↗

arXiv2023

INVE: Interactive Neural Video Editing

We present Interactive Neural Video Editing (INVE), a real-time video editing solution, which can assist the video editing process by consistently propagating sparse frame edits to the entire video clip. Our method is inspired by the recent work on Layered Neural Atlas (LNA). LNA, however, suffers from two major drawbacks: (1) the method is too slow for interactive editing, and (2) it offers insufficient support for some editing use cases, including direct frame editing and rigid texture tracking. To address these challenges we leverage and adopt highly efficient network architectures, powered by hash-grids encoding, to substantially improve processing speed. In addition, we learn bi-directional functions between image-atlas and introduce vectorized editing, which collectively enables a much greater variety of edits in both the atlas and the frames directly. Compared to LNA, our INVE reduces the learning and inference time by a factor of 5, and supports various video editing operations that LNA cannot. We showcase the superiority of INVE over LNA in interactive video editing through a comprehensive quantitative and qualitative analysis, highlighting its numerous advantages and improved performance. For video results, please see https://gabriel-huang.github.io/inve/

cs.CV↗

arXiv2023

Black Hole Accretion with Saturated Magnetic Pressure and Disk Wind

We construct an analytical black hole accretion disk model that incorporates both magnetic pressure and disk wind, which are found to be important from numerical simulations. A saturated magnetic pressure that relates the Alfven velocity with local Keplerian velocity and gas sound speed is assumed in addition to radiation and gas pressures. The mass accretion rate is assumed to have a power-law form in response to mass loss in the wind. We find three sets of self-consistent solutions that are thermally stable and satisfy the model assumptions. At high accretion rates, the disk is geometrically and optically thick, resembling the slim disk solution. At relatively low accretion rates, our model predicts an accretion flow consisting of a geometrically thin and optically thick outer disk (similar to the standard disk), and a geometrically thick and optically thin inner disk (similar to the advection-dominated accretion flow or ADAF). Thus, this is a natural solution for a truncated disk connected with an inner ADAF, which has been proposed to explain some observations. The magnetic pressure plays a more important role than the outflow in shaping the disk structure. The observed disk luminosity tends to saturate around 8 times the Eddington limit, suggesting that supercritical accretion onto black holes can be used for black hole mass estimate, or a standard candle with known black hole masses.

astro-ph.HE↗

arXiv2023

Neural Kernel Surface Reconstruction

We present a novel method for reconstructing a 3D implicit surface from a large-scale, sparse, and noisy point cloud. Our approach builds upon the recently introduced Neural Kernel Fields (NKF) representation. It enjoys similar generalization capabilities to NKF, while simultaneously addressing its main limitations: (a) We can scale to large scenes through compactly supported kernel functions, which enable the use of memory-efficient sparse linear solvers. (b) We are robust to noise, through a gradient fitting solve. (c) We minimize training requirements, enabling us to learn from any dataset of dense oriented points, and even mix training data consisting of objects and scenes at different scales. Our method is capable of reconstructing millions of points in a few seconds, and handling very large scenes in an out-of-core fashion. We achieve state-of-the-art results on reconstruction benchmarks consisting of single objects, indoor scenes, and outdoor scenes.

cs.CV↗

arXiv2023

Equivariant Segre and Verlinde invariants for Quot schemes

The problem of studying the two seemingly unrelated sets of invariants forming the Segre and the Verlinde series has gone through multiple different adaptations including a version for the virtual geometries of Quot schemes on surfaces and Calabi-Yau fourfolds. Our work is the first one to address the equivariant setting for both $\mathbb{C}^2$ and $\mathbb{C}^4$ by examining higher degree contributions which have no compact analogue. (1) For $\mathbb{C}^2$, we work mostly with virtual geometries of Quot schemes. After connecting the equivariant series in degree zero to the existing results of the first author for compact surfaces, we extend the Segre-Verlinde correspondence to all degrees and to the reduced virtual classes. Apart from it, we conjecture an equivariant symmetry between two different Segre series building again on previous work. (2) For $\mathbb{C}^4$, we give further motivation for the definition of the Verlinde series. Based on empirical data and additional structural results, we conjecture the equivariant Segre-Verlinde correspondence and the Segre-Segre symmetry analogous to the one for $\mathbb{C}^2$.

math.AG↗

arXiv2023

X-ray Polarimetry of the accreting pulsar 1A~0535+262 in the supercritical state with PolarLight

The X-ray pulsar 1A 0535+262 exhibited a giant outburst in 2020, offering us a unique opportunity for X-ray polarimetry of an accreting pulsar in the supercritical state. Measurement with PolarLight yielded a non-detection in 3-8 keV; the 99% upper limit of the polarization fraction (PF) is found to be 0.34 averaged over spin phases, or 0.51 based on the rotating vector model. No useful constraint can be placed with phase resolved polarimetry. These upper limits are lower than a previous theoretical prediction of 0.6-0.8, but consistent with those found in other accreting pulsars, like Her X-1, Cen X-3, 4U 1626-67, and GRO J1008-57, which were in the subcritical state, or at least not confidently in the supercritical state, during the polarization measurements. Our results suggest that the relatively low PF seen in accreting pulsars cannot be attributed to the source not being in the supercritical state, but could be a general feature.

astro-ph.HE↗