arXiv subjects

Tian Fang

Publications and source records attributed to Tian Fang.

At least 37 records · Page 2Linked to original sources

arXiv2020

ASLFeat: Learning Local Features of Accurate Shape and Localization

This work focuses on mitigating two limitations in the joint learning of local feature detectors and descriptors. First, the ability to estimate the local shape (scale, orientation, etc.) of feature points is often neglected during dense feature extraction, while the shape-awareness is crucial to acquire stronger geometric invariance. Second, the localization accuracy of detected keypoints is not sufficient to reliably recover camera geometry, which has become the bottleneck in tasks such as 3D reconstruction. In this paper, we present ASLFeat, with three light-weight yet effective modifications to mitigate above issues. First, we resort to deformable convolutional networks to densely estimate and apply local transformation. Second, we take advantage of the inherent feature hierarchy to restore spatial resolution and low-level details for accurate keypoint localization. Finally, we use a peakiness measurement to relate feature responses and derive more indicative detection scores. The effect of each modification is thoroughly studied, and the evaluation is extensively conducted across a variety of practical scenarios. State-of-the-art results are reported that demonstrate the superiority of our methods.

cs.CV↗

arXiv2020

Joint Semantic Segmentation and Boundary Detection using Iterative Pyramid Contexts

In this paper, we present a joint multi-task learning framework for semantic segmentation and boundary detection. The critical component in the framework is the iterative pyramid context module (PCM), which couples two tasks and stores the shared latent semantics to interact between the two tasks. For semantic boundary detection, we propose the novel spatial gradient fusion to suppress nonsemantic edges. As semantic boundary detection is the dual task of semantic segmentation, we introduce a loss function with boundary consistency constraint to improve the boundary pixel accuracy for semantic segmentation. Our extensive experiments demonstrate superior performance over state-of-the-art works, not only in semantic segmentation but also in semantic boundary detection. In particular, a mean IoU score of 81:8% on Cityscapes test set is achieved without using coarse data or any external data for semantic segmentation. For semantic boundary detection, we improve over previous state-of-the-art works by 9.9% in terms of AP and 6:8% in terms of MF(ODS).

cs.CV↗

arXiv2020

BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks

While deep learning has recently achieved great success on multi-view stereo (MVS), limited training data makes the trained model hard to be generalized to unseen scenarios. Compared with other computer vision tasks, it is rather difficult to collect a large-scale MVS dataset as it requires expensive active scanners and labor-intensive process to obtain ground truth 3D structures. In this paper, we introduce BlendedMVS, a novel large-scale dataset, to provide sufficient training ground truth for learning-based MVS. To create the dataset, we apply a 3D reconstruction pipeline to recover high-quality textured meshes from images of well-selected scenes. Then, we render these mesh models to color images and depth maps. To introduce the ambient lighting information during training, the rendered color images are further blended with the input images to generate the training input. Our dataset contains over 17k high-resolution images covering a variety of scenes, including cities, architectures, sculptures and small objects. Extensive experiments demonstrate that BlendedMVS endows the trained model with significantly better generalization ability compared with other MVS datasets. The dataset and pretrained models are available at \url{https://github.com/YoYo000/BlendedMVS}.

cs.CV↗

arXiv2020

KFNet: Learning Temporal Camera Relocalization using Kalman Filtering

Temporal camera relocalization estimates the pose with respect to each video frame in sequence, as opposed to one-shot relocalization which focuses on a still image. Even though the time dependency has been taken into account, current temporal relocalization methods still generally underperform the state-of-the-art one-shot approaches in terms of accuracy. In this work, we improve the temporal relocalization method by using a network architecture that incorporates Kalman filtering (KFNet) for online camera relocalization. In particular, KFNet extends the scene coordinate regression problem to the time domain in order to recursively establish 2D and 3D correspondences for the pose determination. The network architecture design and the loss formulation are based on Kalman filtering in the context of Bayesian learning. Extensive experiments on multiple relocalization benchmarks demonstrate the high accuracy of KFNet at the top of both one-shot and temporal relocalization approaches. Our codes are released at https://github.com/zlthinker/KFNet.

cs.CV↗

arXiv2019

Self-Supervised Learning of Depth and Motion Under Photometric Inconsistency

The self-supervised learning of depth and pose from monocular sequences provides an attractive solution by using the photometric consistency of nearby frames as it depends much less on the ground-truth data. In this paper, we address the issue when previous assumptions of the self-supervised approaches are violated due to the dynamic nature of real-world scenes. Different from handling the noise as uncertainty, our key idea is to incorporate more robust geometric quantities and enforce internal consistency in the temporal image sequence. As demonstrated on commonly used benchmark datasets, the proposed method substantially improves the state-of-the-art methods on both depth and relative pose estimation for monocular image sequences, without adding inference overhead.

cs.CV↗

arXiv2019

Learning Fully Dense Neural Networks for Image Semantic Segmentation

Semantic segmentation is pixel-wise classification which retains critical spatial information. The "feature map reuse" has been commonly adopted in CNN based approaches to take advantage of feature maps in the early layers for the later spatial reconstruction. Along this direction, we go a step further by proposing a fully dense neural network with an encoder-decoder structure that we abbreviate as FDNet. For each stage in the decoder module, feature maps of all the previous blocks are adaptively aggregated to feed-forward as input. On the one hand, it reconstructs the spatial boundaries accurately. On the other hand, it learns more efficiently with the more efficient gradient backpropagation. In addition, we propose the boundary-aware loss function to focus more attention on the pixels near the boundary, which boosts the "hard examples" labeling. We have demonstrated the best performance of the FDNet on the two benchmark datasets: PASCAL VOC 2012, NYUDv2 over previous works when not considering training on other datasets.

cs.CV↗

arXiv2019

ContextDesc: Local Descriptor Augmentation with Cross-Modality Context

Most existing studies on learning local features focus on the patch-based descriptions of individual keypoints, whereas neglecting the spatial relations established from their keypoint locations. In this paper, we go beyond the local detail representation by introducing context awareness to augment off-the-shelf local feature descriptors. Specifically, we propose a unified learning framework that leverages and aggregates the cross-modality contextual information, including (i) visual context from high-level image representation, and (ii) geometric context from 2D keypoint distribution. Moreover, we propose an effective N-pair loss that eschews the empirical hyper-parameter search and improves the convergence. The proposed augmentation scheme is lightweight compared with the raw local feature description, meanwhile improves remarkably on several large-scale benchmarks with diversified scenes, which demonstrates both strong practicality and generalization ability in geometric matching applications.

cs.CV↗

arXiv2019

Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference

Deep learning has recently demonstrated its excellent performance for multi-view stereo (MVS). However, one major limitation of current learned MVS approaches is the scalability: the memory-consuming cost volume regularization makes the learned MVS hard to be applied to high-resolution scenes. In this paper, we introduce a scalable multi-view stereo framework based on the recurrent neural network. Instead of regularizing the entire 3D cost volume in one go, the proposed Recurrent Multi-view Stereo Network (R-MVSNet) sequentially regularizes the 2D cost maps along the depth direction via the gated recurrent unit (GRU). This reduces dramatically the memory consumption and makes high-resolution reconstruction feasible. We first show the state-of-the-art performance achieved by the proposed R-MVSNet on the recent MVS benchmarks. Then, we further demonstrate the scalability of the proposed method on several large-scale scenarios, where previous learned approaches often fail due to the memory constraint. Code is available at https://github.com/YoYo000/MVSNet.

cs.CV↗

arXiv2019

Beyond Photometric Loss for Self-Supervised Ego-Motion Estimation

Accurate relative pose is one of the key components in visual odometry (VO) and simultaneous localization and mapping (SLAM). Recently, the self-supervised learning framework that jointly optimizes the relative pose and target image depth has attracted the attention of the community. Previous works rely on the photometric error generated from depths and poses between adjacent frames, which contains large systematic error under realistic scenes due to reflective surfaces and occlusions. In this paper, we bridge the gap between geometric loss and photometric loss by introducing the matching loss constrained by epipolar geometry in a self-supervised framework. Evaluated on the KITTI dataset, our method outperforms the state-of-the-art unsupervised ego-motion estimation methods by a large margin. The code and data are available at https://github.com/hlzz/DeepMatchVO.

cs.CV↗

arXiv2018

Matchable Image Retrieval by Learning from Surface Reconstruction

Convolutional Neural Networks (CNNs) have achieved superior performance on object image retrieval, while Bag-of-Words (BoW) models with handcrafted local features still dominate the retrieval of overlapping images in 3D reconstruction. In this paper, we narrow down this gap by presenting an efficient CNN-based method to retrieve images with overlaps, which we refer to as the matchable image retrieval problem. Different from previous methods that generates training data based on sparse reconstruction, we create a large-scale image database with rich 3D geometrics and exploit information from surface reconstruction to obtain fine-grained training data. We propose a batched triplet-based loss function combined with mesh re-projection to effectively learn the CNN representation. The proposed method significantly accelerates the image retrieval process in 3D reconstruction and outperforms the state-of-the-art CNN-based and BoW methods for matchable image retrieval. The code and data are available at https://github.com/hlzz/mirror.

cs.CV↗

arXiv2018

GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints

Learned local descriptors based on Convolutional Neural Networks (CNNs) have achieved significant improvements on patch-based benchmarks, whereas not having demonstrated strong generalization ability on recent benchmarks of image-based 3D reconstruction. In this paper, we mitigate this limitation by proposing a novel local descriptor learning approach that integrates geometry constraints from multi-view reconstructions, which benefits the learning process in terms of data generation, data sampling and loss computation. We refer to the proposed descriptor as GeoDesc, and demonstrate its superior performance on various large-scale benchmarks, and in particular show its great success on challenging reconstruction tasks. Moreover, we provide guidelines towards practical integration of learned descriptors in Structure-from-Motion (SfM) pipelines, showing the good trade-off that GeoDesc delivers to 3D reconstruction tasks between accuracy and efficiency.

cs.CV↗

arXiv2018

MVSNet: Depth Inference for Unstructured Multi-view Stereo

We present an end-to-end deep learning architecture for depth map inference from multi-view images. In the network, we first extract deep visual image features, and then build the 3D cost volume upon the reference camera frustum via the differentiable homography warping. Next, we apply 3D convolutions to regularize and regress the initial depth map, which is then refined with the reference image to generate the final output. Our framework flexibly adapts arbitrary N-view inputs using a variance-based cost metric that maps multiple features into one cost feature. The proposed MVSNet is demonstrated on the large-scale indoor DTU dataset. With simple post-processing, our method not only significantly outperforms previous state-of-the-arts, but also is several times faster in runtime. We also evaluate MVSNet on the complex outdoor Tanks and Temples dataset, where our method ranks first before April 18, 2018 without any fine-tuning, showing the strong generalization ability of MVSNet.

cs.CV↗

arXiv2017

Parallel Structure from Motion from Local Increment to Global Averaging

In this paper, we tackle the accurate and consistent Structure from Motion (SfM) problem, in particular camera registration, far exceeding the memory of a single computer in parallel. Different from the previous methods which drastically simplify the parameters of SfM and sacrifice the accuracy of the final reconstruction, we try to preserve the connectivities among cameras by proposing a camera clustering algorithm to divide a large SfM problem into smaller sub-problems in terms of camera clusters with overlapping. We then exploit a hybrid formulation that applies the relative poses from local incremental SfM into a global motion averaging framework and produce accurate and consistent global camera poses. Our scalable formulation in terms of camera clusters is highly applicable to the whole SfM pipeline including track generation, local SfM, 3D point triangulation and bundle adjustment. We are even able to reconstruct the camera poses of a city-scale data-set containing more than one million high-resolution images with superior accuracy and robustness evaluated on benchmark, Internet, and sequential data-sets.

cs.CV↗

arXiv2013

Raman and Photoluminescence Study of Dielectric and Thermal Effects on Atomically Thin MoS2

Atomically thin two-dimensional molybdenum disulfide (MoS2) sheets have attracted much attention due to their potential for future electronic applications. They not only present the best planar electrostatic control in a device, but also lend themselves readily for dielectric engineering. In this work, we experimentally investigated the dielectric effect on the Raman and photoluminescence (PL) spectra of monolayer MoS2 by comparing samples with and without HfO2 on top by atomic layer deposition (ALD). Based on considerations of the thermal, doping, strain and dielectric screening influences, it is found that the red shift in the Raman spectrum largely stems from modulation doping of MoS2 by the ALD HfO2, and the red shift in the PL spectrum is most likely due to strain imparted on MoS2 by HfO2. Our work also suggests that due to the intricate dependence of band structure of monolayer MoS2 on strain, one must be cautious to interpret its Raman and PL spectroscopy.

cond-mat.mtrl-sci↗

arXiv2011

Interband absorption in single layer hexagonal boron nitride

Monolayer of hexagonal boron nitride (h-BN), commonly known as "white graphene" is a promising wide bandgap semiconducting material for deep-ultaviolet optoelectronic devices. In this report, the light absorption of a single layer hexagonal boron nitride is calculated using a tight-binding Hamiltonian. The absorption is found to be monotonically decreasing function of photon energy compared to graphene where absorption coefficient is independent of photon energy and characterized by the effective fine-structure constant.

cond-mat.mes-hall↗

arXiv2011

High field transport in graphene

In this work, high field carrier transport in two dimensional (2D) graphene is investigated. Analytical models are applied to estimate the saturation currents in graphene, based on the high scattering rate of optical phonon emission. Non-equilibrium (hot) phonon effect was studied by Monte Carlo (MC) simulations. MC simulation confirms that hot phonon effects play a dominant role in current saturation in graphene. Current degradation due to elastic scattering events is much smaller compared to the hot phonon effect. Transient phenomenon as such as velocity overshoot was also studied using MC simulation. The simulation results shows promising potential for graphene to be used in high speed electronic devices by shrinking the channel length below 100nm if electrostatic control can be exercised in the absence of a band gap.

cond-mat.mtrl-sci↗

arXiv2011

Unique prospects of graphene-based THz modulators

The modulation depth of 2-D electron gas (2DEG) based THz modulators using AlGaAs/GaAs heterostructures with metal gates is inherently limited to < 30%. The metal gate not only attenuates the THz signal (> 90%) but also severely degrades the modulation depth. The metal losses can be significantly reduced with an alternative material with tunable conductivity. Graphene presents a unique solution to this problem due to its symmetric band structure and extraordinarily high mobility of holes that is comparable to electron mobility in conventional semiconductors. The hole conductivity in graphene can be electrostatically tuned in the graphene-2DEG parallel capacitor configuration, thus more efficiently tuning the THz transmission. In this work, we show that it is possible to achieve a modulation depth of > 90% while simultaneously minimizing signal attenuation to < 5% by tuning the Fermi level at the Dirac point in graphene.

cond-mat.mtrl-sci↗

arXiv2011

Dielectric environment mediated quantum screening of one dimensional electron gas

Relaxing the assumption of "infinite and homogenous background" the dielectric response function of one-dimensional (1D) semiconducting nanowires embedded in a dielectric environment is calculated. It is shown that high-k (higher than semiconductor dielectric constant) dielectric environment reduces the screening by the free carriers inside the nanostructure whereas, low dielectric environment increases the Coulombic interaction between free carriers and enhances the strength of screening function. In long wavelength limit, dielectric screening and collective excitation of electron gas are found to be solely determined by the environment instead of the semiconductor. Behavior of static dielectric function is particularly addressed at a specific wavevector q=2k_F; a wavevector ubiquitously appears in charge transport in nanostructures

cond-mat.mes-hall↗