arXiv subjects

Gang Zhang

Publications and source records attributed to Gang Zhang.

At least 55 records · Page 3Linked to original sources

arXiv2024

LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction

Existing methods enhance open-vocabulary object detection by leveraging the robust open-vocabulary recognition capabilities of Vision-Language Models (VLMs), such as CLIP.However, two main challenges emerge:(1) A deficiency in concept representation, where the category names in CLIP's text space lack textual and visual knowledge.(2) An overfitting tendency towards base categories, with the open vocabulary knowledge biased towards base categories during the transfer from VLMs to detectors.To address these challenges, we propose the Language Model Instruction (LaMI) strategy, which leverages the relationships between visual concepts and applies them within a simple yet effective DETR-like detector, termed LaMI-DETR.LaMI utilizes GPT to construct visual concepts and employs T5 to investigate visual similarities across categories.These inter-category relationships refine concept representation and avoid overfitting to base categories.Comprehensive experiments validate our approach's superior performance over existing methods in the same rigorous setting without reliance on external training resources.LaMI-DETR achieves a rare box AP of 43.4 on OV-LVIS, surpassing the previous best by 7.8 rare box AP.

cs.CV↗

arXiv2024

OVLW-DETR: Open-Vocabulary Light-Weighted Detection Transformer

Open-vocabulary object detection focusing on detecting novel categories guided by natural language. In this report, we propose Open-Vocabulary Light-Weighted Detection Transformer (OVLW-DETR), a deployment friendly open-vocabulary detector with strong performance and low latency. Building upon OVLW-DETR, we provide an end-to-end training recipe that transferring knowledge from vision-language model (VLM) to object detector with simple alignment. We align detector with the text encoder from VLM by replacing the fixed classification layer weights in detector with the class-name embeddings extracted from the text encoder. Without additional fusing module, OVLW-DETR is flexible and deployment friendly, making it easier to implement and modulate. improving the efficiency of interleaved attention computation. Experimental results demonstrate that the proposed approach is superior over existing real-time open-vocabulary detectors on standard Zero-Shot LVIS benchmark. Source code and pre-trained models are available at [https://github.com/Atten4Vis/LW-DETR].

cs.CV↗

arXiv2024

The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Report

This paper provides a comprehensive review of the NTIRE 2024 challenge, focusing on efficient single-image super-resolution (ESR) solutions and their outcomes. The task of this challenge is to super-resolve an input image with a magnification factor of x4 based on pairs of low and corresponding high-resolution images. The primary objective is to develop networks that optimize various aspects such as runtime, parameters, and FLOPs, while still maintaining a peak signal-to-noise ratio (PSNR) of approximately 26.90 dB on the DIV2K_LSDIR_valid dataset and 26.99 dB on the DIV2K_LSDIR_test dataset. In addition, this challenge has 4 tracks including the main track (overall performance), sub-track 1 (runtime), sub-track 2 (FLOPs), and sub-track 3 (parameters). In the main track, all three metrics (ie runtime, FLOPs, and parameter count) were considered. The ranking of the main track is calculated based on a weighted sum-up of the scores of all other sub-tracks. In sub-track 1, the practical runtime performance of the submissions was evaluated, and the corresponding score was used to determine the ranking. In sub-track 2, the number of FLOPs was considered. The score calculated based on the corresponding FLOPs was used to determine the ranking. In sub-track 3, the number of parameters was considered. The score calculated based on the corresponding parameters was used to determine the ranking. RLFN is set as the baseline for efficiency measurement. The challenge had 262 registered participants, and 34 teams made valid submissions. They gauge the state-of-the-art in efficient single-image super-resolution. To facilitate the reproducibility of the challenge and enable other researchers to build upon these findings, the code and the pre-trained model of validated solutions are made publicly available at https://github.com/Amazingren/NTIRE2024_ESR/.

cs.CV↗

arXiv2024

LW-DETR: A Transformer Replacement to YOLO for Real-Time Detection

In this paper, we present a light-weight detection transformer, LW-DETR, which outperforms YOLOs for real-time object detection. The architecture is a simple stack of a ViT encoder, a projector, and a shallow DETR decoder. Our approach leverages recent advanced techniques, such as training-effective techniques, e.g., improved loss and pretraining, and interleaved window and global attentions for reducing the ViT encoder complexity. We improve the ViT encoder by aggregating multi-level feature maps, and the intermediate and final feature maps in the ViT encoder, forming richer feature maps, and introduce window-major feature map organization for improving the efficiency of interleaved attention computation. Experimental results demonstrate that the proposed approach is superior over existing real-time detectors, e.g., YOLO and its variants, on COCO and other benchmark datasets. Code and models are available at (https://github.com/Atten4Vis/LW-DETR).

cs.CV↗

arXiv2024

Real topological phonons in 3D carbon allotropes

There has been a significant focus on real topological systems that enjoy space-time inversion symmetry (PT ) and lack spin-orbit coupling. While the theoretical classification of the real topology has been established, more progress has yet to be made in the materials realization of such real topological systems in three dimensions (3D). To address this crucial issue, by selecting the carbon-based material candidates as targets, we perform high-throughput computing to inspect the real topology in the phonon spectrums of the 3D carbon allotropes in the Samara Carbon Allotrope Database (SACADA). Among 1192 kinds of 3D carbon allotropes, we find 65 real topological systems with a phononic real Chern insulating (PRCI) state, 2 real topological systems with a phononic real nodal line (PRNL) state, 10 real topological systems with a phononic real Dirac point (PRDP) state, and 8 real topological systems with a phononic real triple-point pair (PRTPP) state. This extremely expands the material candidates with real topology, especially for the gapless topological phonons. We exhibit the PRCI, PRNL, PRTPP, and PRDP states of 27-SG. 166-pcu-h, 1081-SG. 194- 4 2T13-CA, 52-SG. 141-gis, and 132-SG. 191-3,4T157 as illustrative examples, and explore the second-order boundary mode, i.e., phononic hinge mode. Among the four examples, the materials 1081-SG. 194-42T13-CA and 52-SG. 141-gis are so ideal that the PRNL and PRTPP in them are well separated from other bands, and the phononic hinge mode can be clearly observed. This study aims to broaden the understanding of 3D topological phonons, and emphasizes the potential of 3D carbon allotropes as a valuable framework for exploring the fascinating physics related to phononic hinge modes and phononic real topology.

cond-mat.mtrl-sci↗

arXiv2024

FastOcc: Accelerating 3D Occupancy Prediction by Fusing the 2D Bird's-Eye View and Perspective View

In autonomous driving, 3D occupancy prediction outputs voxel-wise status and semantic labels for more comprehensive understandings of 3D scenes compared with traditional perception tasks, such as 3D object detection and bird's-eye view (BEV) semantic segmentation. Recent researchers have extensively explored various aspects of this task, including view transformation techniques, ground-truth label generation, and elaborate network design, aiming to achieve superior performance. However, the inference speed, crucial for running on an autonomous vehicle, is neglected. To this end, a new method, dubbed FastOcc, is proposed. By carefully analyzing the network effect and latency from four parts, including the input image resolution, image backbone, view transformation, and occupancy prediction head, it is found that the occupancy prediction head holds considerable potential for accelerating the model while keeping its accuracy. Targeted at improving this component, the time-consuming 3D convolution network is replaced with a novel residual-like architecture, where features are mainly digested by a lightweight 2D BEV convolution network and compensated by integrating the 3D voxel features interpolated from the original image features. Experiments on the Occ3D-nuScenes benchmark demonstrate that our FastOcc achieves state-of-the-art results with a fast inference speed.

cs.CV↗

arXiv2023

3D carbon allotropes: Topological quantum materials with obstructed atomic insulating phases, multiple bulk-boundary correspondences, and real topology

The study of topological phases with unconventional bulk-boundary correspondences and nontrivial real Chern number has garnered significant attention in the topological states of matter. Using the first-principle calculations and theoretical analysis, we perform a high-throughput material screening of the 3D obstructed atomic insulators (OAIs) and 3D real Chern insulators (RCIs) based on the Samara Carbon Allotrope Database (SACADA). Results show that 422 out of 703 3D carbon allotropes are 3D OAIs with multiple bulk-boundary correspondences, including 2D obstructed surface states (OSSs) and 1D hinge states, which are in one dimension and two dimensions lower than the 3D bulk, respectively. The 2D OSSs in these OAIs can be modified when subjected to appropriate boundaries, which benefits the investigation of surface engineering and the development of efficient topological catalysts. These 422 OAIs, which have 2D and 1D boundary states, are excellent platforms for multi-dimensional topological boundaries research. Remarkably, 138 of 422 OAIs are also 3D RCIs, which show a nontrivial real topology in the protection of spacetime inversion symmetry. Our work not only provides a comprehensive list of 3D carbon-based OAIs and RCIs, but also guides their application in various aspects based on multiple bulk-boundary correspondences and real topological phases.

cond-mat.mtrl-sci↗

arXiv2023

Center Focusing Network for Real-Time LiDAR Panoptic Segmentation

LiDAR panoptic segmentation facilitates an autonomous vehicle to comprehensively understand the surrounding objects and scenes and is required to run in real time. The recent proposal-free methods accelerate the algorithm, but their effectiveness and efficiency are still limited owing to the difficulty of modeling non-existent instance centers and the costly center-based clustering modules. To achieve accurate and real-time LiDAR panoptic segmentation, a novel center focusing network (CFNet) is introduced. Specifically, the center focusing feature encoding (CFFE) is proposed to explicitly understand the relationships between the original LiDAR points and virtual instance centers by shifting the LiDAR points and filling in the center points. Moreover, to leverage the redundantly detected centers, a fast center deduplication module (CDM) is proposed to select only one center for each instance. Experiments on the SemanticKITTI and nuScenes panoptic segmentation benchmarks demonstrate that our CFNet outperforms all existing methods by a large margin and is 1.6 times faster than the most efficient method. The code is available at https://github.com/GangZhang842/CFNet.

cs.CV↗

arXiv2023

Sumanene monolayer of pure carbon: a two-dimensional Kagome-analogy lattice with desirable band gap, ultrahigh carrier mobility and strong exciton binding energy

Design and synthesis of novel two-dimensional (2D) materials that possess robust structural stability and unusual physical properties may open up enormous opportunities for device and engineering applications. Herein we propose a 2D sumanene lattice that be regarded as a derivative of the conventional Kagome lattice. Our tight-binding analysis demonstrates sumanene lattice contains two sets of Dirac cones and two sets of flat bands near the Fermi surface, distinctively different from the Kagome lattice. Using first-principles calculations, we theoretically suggest two possible routines for realization of stable 2D sumanene monolayers (named as a phase and b phase), and a-sumanene monolayer can be experimentally synthesized with chemical vapor deposition using C21H12 as a precursor. Small binding energies on Au(111) surface signify the possibility of their peel-off after grown on the noble metal substrate. Importantly, our GW plus Bethe-Salpeter equation calculations demonstrate both monolayers have moderate band gaps (1.94 eV for a) and ultrahigh carrier mobilities (3.4*104 cm2/Vs for a). In particular, a-sumanene monolayer possesses a strong exciton binding energy of 0.73 eV, suggesting potential applications in optics.

cond-mat.mtrl-sci↗

arXiv2023

CEDNet: A Cascade Encoder-Decoder Network for Dense Prediction

Multi-scale features are essential for dense prediction tasks, such as object detection, instance segmentation, and semantic segmentation. The prevailing methods usually utilize a classification backbone to extract multi-scale features and then fuse these features using a lightweight module (e.g., the fusion module in FPN and BiFPN, two typical object detection methods). However, as these methods allocate most computational resources to the classification backbone, the multi-scale feature fusion in these methods is delayed, which may lead to inadequate feature fusion. While some methods perform feature fusion from early stages, they either fail to fully leverage high-level features to guide low-level feature learning or have complex structures, resulting in sub-optimal performance. We propose a streamlined cascade encoder-decoder network, dubbed CEDNet, tailored for dense \mbox{prediction} tasks. All stages in CEDNet share the same encoder-decoder structure and perform multi-scale feature fusion within the decoder. A hallmark of CEDNet is its ability to incorporate high-level features from early stages to guide low-level feature learning in subsequent stages, thereby enhancing the effectiveness of multi-scale feature fusion. We explored three well-known encoder-decoder structures: Hourglass, UNet, and FPN. When integrated into CEDNet, they performed much better than traditional methods that use a pre-designed classification backbone combined with a lightweight fusion module. Extensive experiments on object detection, instance segmentation, and semantic segmentation demonstrated the effectiveness of our method. The code is available at https://github.com/zhanggang001/CEDNet.

cs.CV↗

arXiv2023

HEDNet: A Hierarchical Encoder-Decoder Network for 3D Object Detection in Point Clouds

3D object detection in point clouds is important for autonomous driving systems. A primary challenge in 3D object detection stems from the sparse distribution of points within the 3D scene. Existing high-performance methods typically employ 3D sparse convolutional neural networks with small kernels to extract features. To reduce computational costs, these methods resort to submanifold sparse convolutions, which prevent the information exchange among spatially disconnected features. Some recent approaches have attempted to address this problem by introducing large-kernel convolutions or self-attention mechanisms, but they either achieve limited accuracy improvements or incur excessive computational costs. We propose HEDNet, a hierarchical encoder-decoder network for 3D object detection, which leverages encoder-decoder blocks to capture long-range dependencies among features in the spatial space, particularly for large and distant objects. We conducted extensive experiments on the Waymo Open and nuScenes datasets. HEDNet achieved superior detection accuracy on both datasets than previous state-of-the-art methods with competitive efficiency. The code is available at https://github.com/zhanggang001/HEDNet.

cs.CV↗

arXiv2023

On the $C_4$-isolation number of a graph

Let $C_k$ be the cycle of length $k$. For any graph $G$, a subset $D \subseteq V(G)$ is a $C_k$-isolating set of $G$ if the graph obtained from $G$ by deleting the closed neighbourhood of $D$ contains no $C_k$ as a subgraph. The $C_k$-isolation number of $G$, denoted by $ι(G,C_k)$, is the cardinality of a smallest $C_k$-isolating set of $G$. Borg (2020) and Borg et al. (2022) proved that if $G \ncong C_3$ is a connected graph of order $n$ and size $m$, then $ι(G,C_3) \leq \frac{n}{4}$ and $ι(G,C_3) \leq \frac{m+1}{5}$. Very recently, Bartolo, Borg and Scicluna showed that if $G$ is a connected graph of order $n$ that is not one of the determined nine graphs, then $ι(G,C_4) \leq \frac{n}{5}$. In this paper, we prove that if $G \ncong C_4$ is a connected graph of size $m$, then $ι(G,C_4) \leq \frac{m+1}{6}$, and we characterize the graphs that attain the bound. Moreover, we conjecture that if $G \ncong C_k$ is a connected graph of size $m$, then $ι(G,C_k) \leq \frac{m+1}{k+2}$.

math.CO↗

arXiv2023

Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving

LiDAR and camera are two critical sensors for multi-modal 3D semantic segmentation and are supposed to be fused efficiently and robustly to promise safety in various real-world scenarios. However, existing multi-modal methods face two key challenges: 1) difficulty with efficient deployment and real-time execution; and 2) drastic performance degradation under weak calibration between LiDAR and cameras. To address these challenges, we propose CPGNet-LCF, a new multi-modal fusion framework extending the LiDAR-only CPGNet. CPGNet-LCF solves the first challenge by inheriting the easy deployment and real-time capabilities of CPGNet. For the second challenge, we introduce a novel weak calibration knowledge distillation strategy during training to improve the robustness against the weak calibration. CPGNet-LCF achieves state-of-the-art performance on the nuScenes and SemanticKITTI benchmarks. Remarkably, it can be easily deployed to run in 20ms per frame on a single Tesla V100 GPU using TensorRT TF16 mode. Furthermore, we benchmark performance over four weak calibration levels, demonstrating the robustness of our proposed approach.

cs.CV↗

arXiv2023

Accelerating Vision Transformers Based on Heterogeneous Attention Patterns

Recently, Vision Transformers (ViTs) have attracted a lot of attention in the field of computer vision. Generally, the powerful representative capacity of ViTs mainly benefits from the self-attention mechanism, which has a high computation complexity. To accelerate ViTs, we propose an integrated compression pipeline based on observed heterogeneous attention patterns across layers. On one hand, different images share more similar attention patterns in early layers than later layers, indicating that the dynamic query-by-key self-attention matrix may be replaced with a static self-attention matrix in early layers. Then, we propose a dynamic-guided static self-attention (DGSSA) method where the matrix inherits self-attention information from the replaced dynamic self-attention to effectively improve the feature representation ability of ViTs. On the other hand, the attention maps have more low-rank patterns, which reflect token redundancy, in later layers than early layers. In a view of linear dimension reduction, we further propose a method of global aggregation pyramid (GLAD) to reduce the number of tokens in later layers of ViTs, such as Deit. Experimentally, the integrated compression pipeline of DGSSA and GLAD can accelerate up to 121% run-time throughput compared with DeiT, which surpasses all SOTA approaches.

cs.CV↗

arXiv2023

A Unified Continual Learning Framework with General Parameter-Efficient Tuning

The "pre-training $\rightarrow$ downstream adaptation" presents both new opportunities and challenges for Continual Learning (CL). Although the recent state-of-the-art in CL is achieved through Parameter-Efficient-Tuning (PET) adaptation paradigm, only prompt has been explored, limiting its application to Transformers only. In this paper, we position prompting as one instantiation of PET, and propose a unified CL framework with general PET, dubbed as Learning-Accumulation-Ensemble (LAE). PET, e.g., using Adapter, LoRA, or Prefix, can adapt a pre-trained model to downstream tasks with fewer parameters and resources. Given a PET method, our LAE framework incorporates it for CL with three novel designs. 1) Learning: the pre-trained model adapts to the new task by tuning an online PET module, along with our adaptation speed calibration to align different PET modules, 2) Accumulation: the task-specific knowledge learned by the online PET module is accumulated into an offline PET module through momentum update, 3) Ensemble: During inference, we respectively construct two experts with online/offline PET modules (which are favored by the novel/historical tasks) for prediction ensemble. We show that LAE is compatible with a battery of PET methods and gains strong CL capability. For example, LAE with Adaptor PET surpasses the prior state-of-the-art by 1.3% and 3.6% in last-incremental accuracy on CIFAR100 and ImageNet-R datasets, respectively. Code is available at \url{https://github.com/gqk/LAE}.

cs.CV↗

arXiv2023

New results on the 1-isolation number of graphs without short cycles

Let $G$ be a graph. A subset $D \subseteq V(G)$ is called a 1-isolating set of $G$ if $Δ(G-N[D]) \leq 1$, that is, $G-N[D]$ consists of isolated edges and isolated vertices only. The $1$-isolation number of $G$, denoted by $ι_1(G)$, is the cardinality of a smallest $1$-isolating set of $G$. In this paper, we prove that if $G \notin \{P_3,C_3,C_7,C_{11}\}$ is a connected graph of order $n$ without $6$-cycles, or without induced 5- and 6-cycles, then $ι_1(G) \leq \frac{n}{4}$. Both bounds are sharp.

math.CO↗

arXiv2023

Open-TransMind: A New Baseline and Benchmark for 1st Foundation Model Challenge of Intelligent Transportation

With the continuous improvement of computing power and deep learning algorithms in recent years, the foundation model has grown in popularity. Because of its powerful capabilities and excellent performance, this technology is being adopted and applied by an increasing number of industries. In the intelligent transportation industry, artificial intelligence faces the following typical challenges: few shots, poor generalization, and a lack of multi-modal techniques. Foundation model technology can significantly alleviate the aforementioned issues. To address these, we designed the 1st Foundation Model Challenge, with the goal of increasing the popularity of foundation model technology in traffic scenarios and promoting the rapid development of the intelligent transportation industry. The challenge is divided into two tracks: all-in-one and cross-modal image retrieval. Furthermore, we provide a new baseline and benchmark for the two tracks, called Open-TransMind. According to our knowledge, Open-TransMind is the first open-source transportation foundation model with multi-task and multi-modal capabilities. Simultaneously, Open-TransMind can achieve state-of-the-art performance on detection, classification, and segmentation datasets of traffic scenarios. Our source code is available at https://github.com/Traffic-X/Open-TransMind.

cs.CV↗

arXiv2023

Dual Relation Knowledge Distillation for Object Detection

Knowledge distillation is an effective method for model compression. However, it is still a challenging topic to apply knowledge distillation to detection tasks. There are two key points resulting in poor distillation performance for detection tasks. One is the serious imbalance between foreground and background features, another one is that small object lacks enough feature representation. To solve the above issues, we propose a new distillation method named dual relation knowledge distillation (DRKD), including pixel-wise relation distillation and instance-wise relation distillation. The pixel-wise relation distillation embeds pixel-wise features in the graph space and applies graph convolution to capture the global pixel relation. By distilling the global pixel relation, the student detector can learn the relation between foreground and background features, and avoid the difficulty of distilling features directly for the feature imbalance issue. Besides, we find that instance-wise relation supplements valuable knowledge beyond independent features for small objects. Thus, the instance-wise relation distillation is designed, which calculates the similarity of different instances to obtain a relation matrix. More importantly, a relation filter module is designed to highlight valuable instance relations. The proposed dual relation knowledge distillation is general and can be easily applied for both one-stage and two-stage detectors. Our method achieves state-of-the-art performance, which improves Faster R-CNN based on ResNet50 from 38.4% to 41.6% mAP and improves RetinaNet based on ResNet50 from 37.4% to 40.3% mAP on COCO 2017.

cs.CV↗