arXiv subjects

Guofeng Zhang

Publications and source records attributed to Guofeng Zhang.

At least 109 records · Page 6Linked to original sources

arXiv2024

Benchmarking Neural Radiance Fields for Autonomous Robots: An Overview

Neural Radiance Fields (NeRF) have emerged as a powerful paradigm for 3D scene representation, offering high-fidelity renderings and reconstructions from a set of sparse and unstructured sensor data. In the context of autonomous robotics, where perception and understanding of the environment are pivotal, NeRF holds immense promise for improving performance. In this paper, we present a comprehensive survey and analysis of the state-of-the-art techniques for utilizing NeRF to enhance the capabilities of autonomous robots. We especially focus on the perception, localization and navigation, and decision-making modules of autonomous robots and delve into tasks crucial for autonomous operation, including 3D reconstruction, segmentation, pose estimation, simultaneous localization and mapping (SLAM), navigation and planning, and interaction. Our survey meticulously benchmarks existing NeRF-based methods, providing insights into their strengths and limitations. Moreover, we explore promising avenues for future research and development in this domain. Notably, we discuss the integration of advanced techniques such as 3D Gaussian splatting (3DGS), large language models (LLM), and generative AIs, envisioning enhanced reconstruction efficiency, scene understanding, decision-making capabilities. This survey serves as a roadmap for researchers seeking to leverage NeRFs to empower autonomous robots, paving the way for innovative solutions that can navigate and interact seamlessly in complex environments.

cs.RO↗

arXiv2024

DATAP-SfM: Dynamic-Aware Tracking Any Point for Robust Structure from Motion in the Wild

This paper proposes a concise, elegant, and robust pipeline to estimate smooth camera trajectories and obtain dense point clouds for casual videos in the wild. Traditional frameworks, such as ParticleSfM~\cite{zhao2022particlesfm}, address this problem by sequentially computing the optical flow between adjacent frames to obtain point trajectories. They then remove dynamic trajectories through motion segmentation and perform global bundle adjustment. However, the process of estimating optical flow between two adjacent frames and chaining the matches can introduce cumulative errors. Additionally, motion segmentation combined with single-view depth estimation often faces challenges related to scale ambiguity. To tackle these challenges, we propose a dynamic-aware tracking any point (DATAP) method that leverages consistent video depth and point tracking. Specifically, our DATAP addresses these issues by estimating dense point tracking across the video sequence and predicting the visibility and dynamics of each point. By incorporating the consistent video depth prior, the performance of motion segmentation is enhanced. With the integration of DATAP, it becomes possible to estimate and optimize all camera poses simultaneously by performing global bundle adjustments for point tracking classified as static and visible, rather than relying on incremental camera registration. Extensive experiments on dynamic sequences, e.g., Sintel and TUM RGBD dynamic sequences, and on the wild video, e.g., DAVIS, demonstrate that the proposed method achieves state-of-the-art performance in terms of camera pose estimation even in complex dynamic challenge scenes.

cs.CV↗

arXiv2024

Quantum coherent feedback control of an N-level atom with multiple excitations

The purpose of this paper is to study the dynamics of a quantum coherent feedback network, where an $N$-level atom is coupled with a cavity and the cavity is also coupled with single or multiple parallel waveguides. When the atom is initialized at the highest energy level, it can emit multiple photons into the cavity, and the photons can be further transmitted to the waveguides and re-interact with the cavity quantum electrodynamics (cavity-QED) system. The transmission of photons in the waveguide can construct a feedback channel with a delay determined by the length of the waveguide. We model the dynamics of the atomic and photonic states of the cavity-QED system as a linear control system with delay. By tuning the control parameters such as the coupling strengths among the atom, cavity and waveguide, the eigenstates of the quantum system can be exponentially stable or unstable, and the exponential stability of the linear quantum control system with delay is related with the generation of single- and multi-photon states. Besides, when the cavity-QED system is coupled with multiple parallel waveguides, the emitted photons oscillate among different waveguides and the stability of quantum states is influenced by the feedback loop length and coupling strengths among waveguides.

quant-ph↗

arXiv2024

Quantum Optimal Control Theory for the Shaping of Flying Qubits

The control of flying qubits carried by itinerant photons is ubiquitous in quantum networks. Beside their logical states, the shape of flying qubits must also be tailored for high-efficiency information transmission. In this paper, we introduce quantum optimal control theory to the shaping of flying qubits. Building on the flying-qubit control model established in our previous work, we design objective functionals for the generation of shaped flying qubits under practical constraints on the emitters and couplers. Numerical simulations employing gradient-descent algorithms demonstrate that the optimized control can effectively mitigate unwanted level and photon leakage caused by these non-idealities. Notably, while coherent control offers limited shaping capacity with a fixed coupler, it can significantly enhance the shaping performance when combined with a tunable coupler that has restricted tunability. The proposed optimal control framework provides a systematic approach to achieving high-quality control of flying qubits using realistic quantum devices.

quant-ph↗

arXiv2024

On Metzler positive systems on hypergraphs

In graph-theoretical terms, an edge in a graph connects two vertices while a hyperedge of a hypergraph connects any more than one vertices. If the hypergraph's hyperedges further connect the same number of vertices, it is said to be uniform. In algebraic graph theory, a graph can be characterized by an adjacency matrix, and similarly, a uniform hypergraph can be characterized by an adjacency tensor. This similarity enables us to extend existing tools of matrix analysis for studying dynamical systems evolving on graphs to the study of a class of polynomial dynamical systems evolving on hypergraphs utilizing the properties of tensors. To be more precise, in this paper, we first extend the concept of a Metzler matrix to a Metzler tensor and then describe some useful properties of such tensors. Next, we focus on positive systems on hypergraphs associated with Metzler tensors. More importantly, we design control laws to stabilize the origin of this class of Metzler positive systems on hypergraphs. In the end, we apply our findings to two classic dynamical systems: a higher-order Lotka-Volterra population dynamics system and a higher-order SIS epidemic dynamic process. The corresponding novel stability results are accompanied by ample numerical examples.

eess.SY↗

arXiv2024

XRDSLAM: A Flexible and Modular Framework for Deep Learning based SLAM

In this paper, we propose a flexible SLAM framework, XRDSLAM. It adopts a modular code design and a multi-process running mechanism, providing highly reusable foundational modules such as unified dataset management, 3d visualization, algorithm configuration, and metrics evaluation. It can help developers quickly build a complete SLAM system, flexibly combine different algorithm modules, and conduct standardized benchmarking for accuracy and efficiency comparison. Within this framework, we integrate several state-of-the-art SLAM algorithms with different types, including NeRF and 3DGS based SLAM, and even odometry or reconstruction algorithms, which demonstrates the flexibility and extensibility. We also conduct a comprehensive comparison and evaluation of these integrated algorithms, analyzing the characteristics of each. Finally, we contribute all the code, configuration and data to the open-source community, which aims to promote the widespread research and development of SLAM technology within the open-source ecosystem.

cs.CV↗

arXiv2024

DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion

Diffusion-based methods have achieved remarkable achievements in 2D image or 3D object generation, however, the generation of 3D scenes and even $360^{\circ}$ images remains constrained, due to the limited number of scene datasets, the complexity of 3D scenes themselves, and the difficulty of generating consistent multi-view images. To address these issues, we first establish a large-scale panoramic video-text dataset containing millions of consecutive panoramic keyframes with corresponding panoramic depths, camera poses, and text descriptions. Then, we propose a novel text-driven panoramic generation framework, termed DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. Specifically, benefiting from the powerful generative capabilities of stable diffusion, we fine-tune a single-view text-to-panorama diffusion model with LoRA on the established panoramic video-text dataset. We further design a spherical epipolar-aware multi-view diffusion model to ensure the multi-view consistency of the generated panoramic images. Extensive experiments demonstrate that DiffPano can generate scalable, consistent, and diverse panoramic images with given unseen text descriptions and camera poses.

cs.CV↗

arXiv2024

LiVisSfM: Accurate and Robust Structure-from-Motion with LiDAR and Visual Cues

This paper presents an accurate and robust Structure-from-Motion (SfM) pipeline named LiVisSfM, which is an SfM-based reconstruction system that fully combines LiDAR and visual cues. Unlike most existing LiDAR-inertial odometry (LIO) and LiDAR-inertial-visual odometry (LIVO) methods relying heavily on LiDAR registration coupled with Inertial Measurement Unit (IMU), we propose a LiDAR-visual SfM method which innovatively carries out LiDAR frame registration to LiDAR voxel map in a Point-to-Gaussian residual metrics, combined with a LiDAR-visual BA and explicit loop closure in a bundle optimization way to achieve accurate and robust LiDAR pose estimation without dependence on IMU incorporation. Besides, we propose an incremental voxel updating strategy for efficient voxel map updating during the process of LiDAR frame registration and LiDAR-visual BA optimization. Experiments demonstrate the superior effectiveness of our LiVisSfM framework over state-of-the-art LIO and LIVO works on more accurate and robust LiDAR pose recovery and dense point cloud reconstruction of both public KITTI benchmark and a variety of self-captured dataset.

cs.CV↗

arXiv2024

BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation

Event cameras provide high temporal precision, low data rates, and high dynamic range visual perception, which are well-suited for optical flow estimation. While data-driven optical flow estimation has obtained great success in RGB cameras, its generalization performance is seriously hindered in event cameras mainly due to the limited and biased training data. In this paper, we present a novel simulator, BlinkSim, for the fast generation of large-scale data for event-based optical flow. BlinkSim incorporates a configurable rendering engine alongside an event simulation suite. By leveraging the wealth of current 3D assets, the rendering engine enables us to automatically build up thousands of scenes with different objects, textures, and motion patterns and render very high-frequency images for realistic event data simulation. Based on BlinkSim, we construct a large training dataset and evaluation benchmark BlinkFlow that contains sufficient, diversiform, and challenging event data with optical flow ground truth. Experiments show that BlinkFlow improves the generalization performance of state-of-the-art methods by more than 40\% on average and up to 90\%. Moreover, we further propose an Event-based optical Flow transFormer (E-FlowFormer) architecture. Powered by our BlinkFlow, E-FlowFormer outperforms the SOTA methods by up to 91\% on the MVSEC dataset and 14\% on the DSEC dataset and presents the best generalization performance. The source code and data are available at https://zju3dv.github.io/blinkflow/.

cs.CV↗

arXiv2024

StreetSurfGS: Scalable Urban Street Surface Reconstruction with Planar-based Gaussian Splatting

Reconstructing urban street scenes is crucial due to its vital role in applications such as autonomous driving and urban planning. These scenes are characterized by long and narrow camera trajectories, occlusion, complex object relationships, and data sparsity across multiple scales. Despite recent advancements, existing surface reconstruction methods, which are primarily designed for object-centric scenarios, struggle to adapt effectively to the unique characteristics of street scenes. To address this challenge, we introduce StreetSurfGS, the first method to employ Gaussian Splatting specifically tailored for scalable urban street scene surface reconstruction. StreetSurfGS utilizes a planar-based octree representation and segmented training to reduce memory costs, accommodate unique camera characteristics, and ensure scalability. Additionally, to mitigate depth inaccuracies caused by object overlap, we propose a guided smoothing strategy within regularization to eliminate inaccurate boundary points and outliers. Furthermore, to address sparse views and multi-scale challenges, we use a dual-step matching strategy that leverages adjacent and long-term information. Extensive experiments validate the efficacy of StreetSurfGS in both novel view synthesis and surface reconstruction.

cs.CV↗

arXiv2024

Quantum coherent and measurement feedback control based on atoms coupled with a semi-infinite waveguide

In this paper, we show that quantum feedback control may be applied to generate desired states for atomic and photonic systems based on a semi-infinite waveguide coupled with multiple two-level atoms. In this set-up, an initially excited atom can emit one photon into the waveguide, which can be reflected by the terminal mirror or other atoms to establish different feedback loops via the coherent interactions between the atom and photon. When there are at most two excitations in the waveguide quantum electrodynamics (waveguide QED) system, the evolution of quantum states can be interpreted using random graph theory. While this process is influenced by the environment, and we clarify that the environment-induced dynamics can be eliminated by measurement-based feedback control or coherent drives. Thus, in the open system atom-waveguide interactions, measurement-based feedback can modulate the final steady quantum state, while simultaneously, the homodyne detection noise in the measurement process can induce oscillations, which is treated by the coherent feedback designs.

quant-ph↗

arXiv2024

SplatLoc: 3D Gaussian Splatting-based Visual Localization for Augmented Reality

Visual localization plays an important role in the applications of Augmented Reality (AR), which enable AR devices to obtain their 6-DoF pose in the pre-build map in order to render virtual content in real scenes. However, most existing approaches can not perform novel view rendering and require large storage capacities for maps. To overcome these limitations, we propose an efficient visual localization method capable of high-quality rendering with fewer parameters. Specifically, our approach leverages 3D Gaussian primitives as the scene representation. To ensure precise 2D-3D correspondences for pose estimation, we develop an unbiased 3D scene-specific descriptor decoder for Gaussian primitives, distilled from a constructed feature volume. Additionally, we introduce a salient 3D landmark selection algorithm that selects a suitable primitive subset based on the saliency score for localization. We further regularize key Gaussian primitives to prevent anisotropic effects, which also improves localization performance. Extensive experiments on two widely used datasets demonstrate that our method achieves superior or comparable rendering and localization performance to state-of-the-art implicit-based visual localization approaches. Project page: \href{https://zju3dv.github.io/splatloc}{https://zju3dv.github.io/splatloc}.

cs.CV↗

arXiv2024

GigaGS: Scaling up Planar-Based 3D Gaussians for Large Scene Surface Reconstruction

3D Gaussian Splatting (3DGS) has shown promising performance in novel view synthesis. Previous methods adapt it to obtaining surfaces of either individual 3D objects or within limited scenes. In this paper, we make the first attempt to tackle the challenging task of large-scale scene surface reconstruction. This task is particularly difficult due to the high GPU memory consumption, different levels of details for geometric representation, and noticeable inconsistencies in appearance. To this end, we propose GigaGS, the first work for high-quality surface reconstruction for large-scale scenes using 3DGS. GigaGS first applies a partitioning strategy based on the mutual visibility of spatial regions, which effectively grouping cameras for parallel processing. To enhance the quality of the surface, we also propose novel multi-view photometric and geometric consistency constraints based on Level-of-Detail representation. In doing so, our method can reconstruct detailed surface structures. Comprehensive experiments are conducted on various datasets. The consistent improvement demonstrates the superiority of GigaGS.

cs.CV↗

arXiv2024

Noise Suppression via Coherent Quantum Feedback: a Schr{ö}dinger Picture Approach

In this article, we explore the possibility of achieving noise suppression for finite-dimensional quantum systems through coherent feedback. For a quantum plant which is expected to evolve according to a target trajectory, noise effect potentially deforms the plant state trajectory from the desired one. It is then hoped that a coherent feedback protocol can be designed that counteracts noise. In terms of coping with transient noise, we present several conditions on coherent feedback protocols under which noise-affected trajectories can be driven back towards desired ones asymptotically. As for rejecting persistent noise, conditions on protocols are given which ensure that the error between the target and feedback-corrected trajectories in the long-time limit can be effectively suppressed. Moreover, a possible construction of coherent feedback protocols which satisfies the given conditions is provided. Our theoretical results are illustrated by an example which involves a two-qubit plant and a two-level controller.

quant-ph↗

arXiv2024

MoManifold: Learning to Measure 3D Human Motion via Decoupled Joint Acceleration Manifolds

Incorporating temporal information effectively is important for accurate 3D human motion estimation and generation which have wide applications from human-computer interaction to AR/VR. In this paper, we present MoManifold, a novel human motion prior, which models plausible human motion in continuous high-dimensional motion space. Different from existing mathematical or VAE-based methods, our representation is designed based on the neural distance field, which makes human dynamics explicitly quantified to a score and thus can measure human motion plausibility. Specifically, we propose novel decoupled joint acceleration manifolds to model human dynamics from existing limited motion data. Moreover, we introduce a novel optimization method using the manifold distance as guidance, which facilitates a variety of motion-related tasks. Extensive experiments demonstrate that MoManifold outperforms existing SOTAs as a prior in several downstream tasks such as denoising real-world human mocap data, recovering human motion from partial 3D observations, mitigating jitters for SMPL-based pose estimators, and refining the results of motion in-betweening.

cs.CV↗

arXiv2024

NOVUM: Neural Object Volumes for Robust Object Classification

Discriminative models for object classification typically learn image-based representations that do not capture the compositional and 3D nature of objects. In this work, we show that explicitly integrating 3D compositional object representations into deep networks for image classification leads to a largely enhanced generalization in out-of-distribution scenarios. In particular, we introduce a novel architecture, referred to as NOVUM, that consists of a feature extractor and a neural object volume for every target object class. Each neural object volume is a composition of 3D Gaussians that emit feature vectors. This compositional object representation allows for a highly robust and fast estimation of the object class by independently matching the features of the 3D Gaussians of each category to features extracted from an input image. Additionally, the object pose can be estimated via inverse rendering of the corresponding neural object volume. To enable the classification of objects, the neural features at each 3D Gaussian are trained discriminatively to be distinct from (i) the features of 3D Gaussians in other categories, (ii) features of other 3D Gaussians of the same object, and (iii) the background features. Our experiments show that NOVUM offers intriguing advantages over standard architectures due to the 3D compositional structure of the object representation, namely: (1) An exceptional robustness across a spectrum of real-world and synthetic out-of-distribution shifts and (2) an enhanced human interpretability compared to standard models, all while maintaining real-time inference and a competitive accuracy on in-distribution data.

cs.CV↗

arXiv2024

D$^3$FlowSLAM: Self-Supervised Dynamic SLAM with Flow Motion Decomposition and DINO Guidance

In this paper, we introduce a self-supervised deep SLAM method that robustly operates in dynamic scenes while accurately identifying dynamic components. Our method leverages a dual-flow representation for static flow and dynamic flow, facilitating effective scene decomposition in dynamic environments. We propose a dynamic update module based on this representation and develop a dense SLAM system that excels in dynamic scenarios. In addition, we design a self-supervised training scheme using DINO as a prior, enabling label-free training. Our method achieves superior accuracy compared to other self-supervised methods. It also matches or even surpasses the performance of existing supervised methods in some cases. All code and data will be made publicly available upon acceptance.

cs.CV↗

arXiv2024

Closed-Loop Designed Open-Loop Control of Quantum Systems: An Error Analysis

Quantum Lyapunov control, an important class of quantum control methods, aims at generating converging dynamics guided by Lyapunov-based theoretical tools. However, unlike the case of classical systems, disturbance caused by quantum measurement hinders direct and exact realization of the theoretical feedback dynamics designed with Lyapunov theory. Regarding this issue, the idea of closed-loop designed open-loop control has been mentioned in literature, which means to design the closed-loop dynamics theoretically, simulate the closed-loop system, generate control pulses based on simulation and apply them to the real plant in an open-loop fashion. Based on bilinear quantum control model, we analyze in this article the error, i.e., difference between the theoretical and real systems' time-evolved states, incurred by the procedures of closed-loop designed open-loop control. It is proved that the error at an arbitrary time converges to a unitary transformation of initialization error as the number of simulation grids between 0 and that time tends to infinity. Moreover, it is found that once the simulation accuracy reaches a certain level, adopting more accurate (thus possibly more expensive) numerical simulation methods does not efficiently improve convergence. We also present an upper bound on the error norm and an example to illustrate our results.

quant-ph↗