arXiv subjects

Zhiyong Li

Publications and source records attributed to Zhiyong Li.

51 records · Page 3Linked to original sources

arXiv2024

EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving

This paper introduces the task of Auditory Referring Multi-Object Tracking (AR-MOT), which dynamically tracks specific objects in a video sequence based on audio expressions and appears as a challenging problem in autonomous driving. Due to the lack of semantic modeling capacity in audio and video, existing works have mainly focused on text-based multi-object tracking, which often comes at the cost of tracking quality, interaction efficiency, and even the safety of assistance systems, limiting the application of such methods in autonomous driving. In this paper, we delve into the problem of AR-MOT from the perspective of audio-video fusion and audio-video tracking. We put forward EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers. The dual streams are intertwined with our Bidirectional Frequency-domain Cross-attention Fusion Module (Bi-FCFM), which bidirectionally fuses audio and video features from both frequency- and spatiotemporal domains. Moreover, we propose the Audio-visual Contrastive Tracking Learning (ACTL) regime to extract homogeneous semantic features between expressions and visual objects by learning homogeneous features between different audio and video objects effectively. Aside from the architectural design, we establish the first set of large-scale AR-MOT benchmarks, including Echo-KITTI, Echo-KITTI+, and Echo-BDD. Extensive experiments on the established benchmarks demonstrate the effectiveness of the proposed EchoTrack and its components. The source code and datasets are available at https://github.com/lab206/EchoTrack.

cs.CV↗

arXiv2024

AdaptiveClick: Clicks-aware Transformer with Adaptive Focal Loss for Interactive Image Segmentation

Interactive Image Segmentation (IIS) has emerged as a promising technique for decreasing annotation time. Substantial progress has been made in pre- and post-processing for IIS, but the critical issue of interaction ambiguity, notably hindering segmentation quality, has been under-researched. To address this, we introduce AdaptiveClick -- a click-aware transformer incorporating an adaptive focal loss that tackles annotation inconsistencies with tools for mask- and pixel-level ambiguity resolution. To the best of our knowledge, AdaptiveClick is the first transformer-based, mask-adaptive segmentation framework for IIS. The key ingredient of our method is the Click-Aware Mask-adaptive transformer Decoder (CAMD), which enhances the interaction between click and image features. Additionally, AdaptiveClick enables pixel-adaptive differentiation of hard and easy samples in the decision space, independent of their varying distributions. This is primarily achieved by optimizing a generalized Adaptive Focal Loss (AFL) with a theoretical guarantee, where two adaptive coefficients control the ratio of gradient values for hard and easy pixels. Our analysis reveals that the commonly used Focal and BCE losses can be considered special cases of the proposed AFL. With a plain ViT backbone, extensive experimental results on nine datasets demonstrate the superiority of AdaptiveClick compared to state-of-the-art methods. The source code is publicly available at https://github.com/lab206/AdaptiveClick.

cs.CV↗

arXiv2023

Large-area, freestanding single-crystal gold of single nanometer thickness

Two-dimensional single-crystal metals are highly sought after for next-generation technologies. Here, we report large-area (>10^4 μm2), single-crystal two-dimensional gold with thicknesses down to a single-nanometer level, employing an atomic-level-precision chemical etching approach. The ultrathin thickness and single-crystal quality endow two-dimensional gold with unique properties including significantly quantum-confinement-augmented optical nonlinearity, low sheet resistance, high transparency and excellent mechanical flexibility. By patterning the two-dimensional gold into nanoribbon arrays, extremely-confined near-infrared plasmonic resonances are further demonstrated with quality factors up to 5. The freestanding nature of two-dimensional gold allows its straightforward manipulation and transfer-printing for integration with other structures. The developed two-dimensional gold provides an emerging platform for fundamental studies in various disciplines and opens up new opportunities for applications in high-performance ultrathin optoelectronic, photonic and quantum devices.

physics.optics↗

arXiv2023

OriWheelBot: An origami-wheeled robot

Origami-inspired robots with multiple advantages, such as being lightweight, requiring less assembly, and exhibiting exceptional deformability, have received substantial and sustained attention. However, the existing origami-inspired robots are usually of limited functionalities and developing feature-rich robots is very challenging. Here, we report an origami-wheeled robot (OriWheelBot) with variable width and outstanding sand walking versatility. The OriWheelBot's ability to adjust wheel width over obstacles is achieved by origami wheels made of Miura origami. An improved version, called iOriWheelBot, is also developed to automatically judge the width of the obstacles. Three actions, namely direct pass, variable width pass, and direct return, will be carried out depending on the width of the channel between the obstacles. We have identified two motion mechanisms, i.e., sand-digging and sand-pushing, with the latter being more conducive to walking on the sand. We have systematically examined numerous sand walking characteristics, including carrying loads, climbing a slope, walking on a slope, and navigating sand pits, small rocks, and sand traps. The OriWheelBot can change its width by 40%, has a loading-carrying ratio of 66.7% on flat sand and can climb a 17-degree sand incline. The OriWheelBot can be useful for planetary subsurface exploration and disaster area rescue.

cs.RO↗

arXiv2023

Bi-Mapper: Holistic BEV Semantic Mapping for Autonomous Driving

A semantic map of the road scene, covering fundamental road elements, is an essential ingredient in autonomous driving systems. It provides important perception foundations for positioning and planning when rendered in the Bird's-Eye-View (BEV). Currently, the prior knowledge of hypothetical depth can guide the learning of translating front perspective views into BEV directly with the help of calibration parameters. However, it suffers from geometric distortions in the representation of distant objects. In addition, another stream of methods without prior knowledge can learn the transformation between front perspective views and BEV implicitly with a global view. Considering that the fusion of different learning methods may bring surprising beneficial effects, we propose a Bi-Mapper framework for top-down road-scene semantic understanding, which incorporates a global view and local prior knowledge. To enhance reliable interaction between them, an asynchronous mutual learning strategy is proposed. At the same time, an Across-Space Loss (ASL) is designed to mitigate the negative impact of geometric distortions. Extensive results on nuScenes and Cam2BEV datasets verify the consistent effectiveness of each module in the proposed Bi-Mapper framework. Compared with exiting road mapping networks, the proposed Bi-Mapper achieves 2.1% higher IoU on the nuScenes dataset. Moreover, we verify the generalization performance of Bi-Mapper in a real-world driving scenario. The source code is publicly available at https://github.com/lynn-yu/Bi-Mapper.

cs.CV↗

arXiv2023

SSD-MonoDETR: Supervised Scale-aware Deformable Transformer for Monocular 3D Object Detection

Transformer-based methods have demonstrated superior performance for monocular 3D object detection recently, which aims at predicting 3D attributes from a single 2D image. Most existing transformer-based methods leverage both visual and depth representations to explore valuable query points on objects, and the quality of the learned query points has a great impact on detection accuracy. Unfortunately, existing unsupervised attention mechanisms in transformers are prone to generate low-quality query features due to inaccurate receptive fields, especially on hard objects. To tackle this problem, this paper proposes a novel "Supervised Scale-aware Deformable Attention" (SSDA) for monocular 3D object detection. Specifically, SSDA presets several masks with different scales and utilizes depth and visual features to adaptively learn a scale-aware filter for object query augmentation. Imposing the scale awareness, SSDA could well predict the accurate receptive field of an object query to support robust query feature generation. Aside from this, SSDA is assigned with a Weighted Scale Matching (WSM) loss to supervise scale prediction, which presents more confident results as compared to the unsupervised attention mechanisms. Extensive experiments on the KITTI and Waymo Open datasets demonstrate that SSDA significantly improves the detection accuracy, especially on moderate and hard objects, yielding state-of-the-art performance as compared to the existing approaches. Our code will be made publicly available at https://github.com/mikasa3lili/SSD-MonoDETR.

cs.CV↗

arXiv2023

Contrast-augmented Diffusion Model with Fine-grained Sequence Alignment for Markup-to-Image Generation

The recently rising markup-to-image generation poses greater challenges as compared to natural image generation, due to its low tolerance for errors as well as the complex sequence and context correlations between markup and rendered image. This paper proposes a novel model named "Contrast-augmented Diffusion Model with Fine-grained Sequence Alignment" (FSA-CDM), which introduces contrastive positive/negative samples into the diffusion model to boost performance for markup-to-image generation. Technically, we design a fine-grained cross-modal alignment module to well explore the sequence similarity between the two modalities for learning robust feature representations. To improve the generalization ability, we propose a contrast-augmented diffusion model to explicitly explore positive and negative samples by maximizing a novel contrastive variational objective, which is mathematically inferred to provide a tighter bound for the model's optimization. Moreover, the context-aware cross attention module is developed to capture the contextual information within markup language during the denoising process, yielding better noise prediction results. Extensive experiments are conducted on four benchmark datasets from different domains, and the experimental results demonstrate the effectiveness of the proposed components in FSA-CDM, significantly exceeding state-of-the-art performance by about 2%-12% DTW improvements. The code will be released at https://github.com/zgj77/FSACDM.

cs.CV↗

arXiv2022

Photonic sampled and quantized analog-to-digital converters on thin-film lithium niobate platform

In this paper, an on-chip photonic sampled and quantized analog-to-digital converter (ADC) on thin-film lithium niobate platform is experimentally demonstrated. Using two phase modulators as a sampler and a 5$\times$5 multimode interference (MMI) coupler as a quantizer, an 1 GHz sinusoidal analog input signal was successfully converted to a digitized output with a 20 GSample/s sampling rate. To evaluate the system performance, the quantization curves together with the transfer function of the ADC were measured. The experimental effective number of bits (ENOB) was 3.17 bit. The demonstrated device is capable of operating at a high frequency up to 70 GHz, making it a promising solution for on-chip ultra-high speed analog-to-digital conversion.

physics.optics↗

arXiv2022

Electrical manipulation of plasmon-phonon polaritons in heterostructures of graphene on biaxial crystals

Phonon polaritons in natural anisotropic crystals hold great promise for infrared nano-optics. However, the direct electrical control of these polaritons is difficult, preventing the development of active polaritonic devices. Here we propose the heterostructures of graphene on a biaxial crystal (α-phase molybdenum trioxide) slab and theoretically study the hybridized plasmon-phonon polaritons with dependence on the Fermi level of graphene from three aspects: dispersion relationships, iso-frequency contours, and the quantum spin Hall effects. We demonstrate the distinct wavelength tunability of the plasmon-phonon polaritons modes and the optical topologic transitions from open (hyperbolic) to closed (bow-tie-like) iso-frequency contours as the increase of the Fermi level of graphene. Furthermore, we observe the tunable quantum spin Hall effects of the plasmon-phonon polaritons, manifesting propagation direction switching by the Fermi level tuning of the graphene. Our findings open opportunities for novel electrically tunable polaritonic devices and programmable quantum optical networks.

cond-mat.mes-hall↗

arXiv2022

STURE: Spatial-Temporal Mutual Representation Learning for Robust Data Association in Online Multi-Object Tracking

Online multi-object tracking (MOT) is a longstanding task for computer vision and intelligent vehicle platform. At present, the main paradigm is tracking-by-detection, and the main difficulty of this paradigm is how to associate current candidate detections with historical tracklets. However, in the MOT scenarios, each historical tracklet is composed of an object sequence, while each candidate detection is just a flat image, which lacks temporal features of the object sequence. The feature difference between current candidate detections and historical tracklets makes the object association much harder. Therefore, we propose a Spatial-Temporal Mutual Representation Learning (STURE) approach which learns spatial-temporal representations between current candidate detections and historical sequences in a mutual representation space. For historical trackelets, the detection learning network is forced to match the representations of sequence learning network in a mutual representation space. The proposed approach is capable of extracting more distinguishing detection and sequence representations by using various designed losses in object association. As a result, spatial-temporal feature is learned mutually to reinforce the current detection features, and the feature difference can be relieved. To prove the robustness of the STURE, it is applied to the public MOT challenge benchmarks and performs well compared with various state-of-the-art online MOT trackers based on identity-preserving metrics.

cs.CV↗

arXiv2022

Energy-efficient Dense DNN Acceleration with Signed Bit-slice Architecture

As the number of deep neural networks (DNNs) to be executed on a mobile system-on-chip (SoC) increases, the mobile SoC suffers from the real-time DNN acceleration within its limited hardware resources and power budget. Although the previous mobile neural processing units (NPUs) take advantage of low-bit computing and exploitation of the sparsity, it is incapable of accelerating high-precision and dense DNNs. This paper proposes energy-efficient signed bit-slice architecture which accelerates both high-precision and dense DNNs by exploiting a large number of zero values of signed bit-slices. Proposed signed bit-slice representation (SBR) changes signed $1111_{2}$ bit-slice to $0000_{2}$ by borrowing a $1$ value from its lower order of bit-slice. As a result, it generates a large number of zero bit-slices even in dense DNNs. Moreover, it balances the positive and negative values of 2's complement data, allowing bit-slice based output speculation which pre-computes high order of bit-slices and skips the remaining dense low order of bit-slices. The signed bit-slice architecture compresses and skips the zero input signed bit-slices, and the zero skipping unit also supports the output skipping by masking the speculated inputs as zero. Additionally, the heterogeneous network-on-chip (NoC) benefits the exploitation of data reusability and reduction of transmission bandwidth. The paper introduces a specialized instruction set architecture (ISA) and a hierarchical instruction decoder for the control of the signed bit-slice architecture. Finally, the signed bit-slice architecture outperforms the previous bit-slice accelerator, Bit-fusion, over $\times3.65$ higher area-efficiency, $\times3.88$ higher energy-efficiency, and $\times5.35$ higher throughput.

cs.AR↗

arXiv2021

A Spontaneously Formed Plasmonic-MoTe2 Hybrid Platform for Ultrasensitive Raman Enhancement

To develop highly sensitive, stable and repeatable surface-enhanced Raman scattering (SERS) substrates is crucial for analytical detection, which is a challenge for traditional metallic structures. Herein, by taking advantage of the high surface activity of 1T' transition metal telluride, we have fabricated high-density gold nanoparticles (AuNPs) that are spontaneously in-situ prepared on the 1T' MoTe2 atomic layers via a facile method, forming a plasmonic-2D material hybrid SERS substrate. This AuNP formation is unique to the 1T' phase, which is repressed in 2H MoTe2 with less surface activity. The hybrid structure generates coupling effects of electromagnetic and chemical enhancements, as well as excellent molecule adsorption, leading to the ultrasensitive (4*10^-17 M) and reproducible detection. Additionally, the immense fluorescence and photobleaching phenomena are mostly avoided. Flexible SERS tapes have been demonstrated in practical applications. Our approach facilitates the ultrasensitive SERS detection by a facile method, as well as the better mechanistic understanding of SERS beyond plasmonic effects.

physics.optics↗

arXiv2021

Silicon rich nitride-lithium niobate on insulator platform for photonics integrated circuits

In this paper, a silicon rich nitride-lithium niobate on insulator (SRN-LNOI) hybrid platform with waveguides and several key components is proposed. The propagation loss of the silicon rich nitride-lithium niobate rib-loaded waveguide (1 um x 300 nm) is 2.6 dB/cm at 1550 nm. Passive devices, including adiabatic power splitters, multimode interferometer based splitters, asymmetrical Mach-Zehnder interferometers and Bragg grating filters are fully designed and characterized. Moreover, we report the first demonstration of Mach-Zehnder modulators based on LNOI with high-speed modulation up to 120 GBaud without digital compensation. Hence, the proposed platform enables high performance passive and active devices with low loss, high integration density and complementary metal-oxide-semiconductor technology (CMOS) compatibility, making it a promising candidate for emerging photonics integrated circuits.

physics.app-ph↗

arXiv2018

Non-cooperative game approach for task offloading in edge clouds

Task offloading provides a promising way to enhance the capability of the mobile terminal (also called terminal user) that is distributed on network edge and communicates edge clouds with wireless. Generally, there are multiple edge cloud nodes with distinct processing capability in a geographic area, which can offer computing service for various terminal users. Furthermore, the terminal users are competitive and selfish, i.e., each user takes into account only maximizing her own profit, while conducting task offloading strategies. In this paper, we focus on the resource management optimization for edge clouds, and formulate the problem of resource competition among terminal users as a non-cooperative game, in which the terminal user who acts as the player always pursues the minimization of the expected response time for her tasks by optimizing allocation strategies. We present the utility function of the user with queuing theory, and then prove the existence of Nash equilibrium for the formulated game. Using the concept of Nash bargaining solution to calculate the optimal task offloading scheme for the user, we propose a distributed task offloading algorithm with low computation complexity. The results of simulated experiments demonstrate that our method can quickly reach the Nash equilibrium point, and deliver satisfying performance at the expected response time of the user's tasks.

cs.GT↗

arXiv2009

Plasmonic resonances and hot spots in Ag octopods

New type of plasmonic nanoparticles - silver octopods that can be synthesized with a variety of shapes - have been demonstrated to show versatile optical response using the discrete dipole approximation. The octopods show a complex behavior at optical (visible, IR) wavelengths, with three major resonances that can be tuned up to a desired response that makes them especially attractive to use in e.g. high-performance surface enhanced Raman (SERS) detectors. The excited resonant modes strongly depend on the geometrical parameters of the stars, while dependence on their orientation with respect to an incident radiation is moderate, owing to cubic symmetry. The field "hot spots" are formed with the local field enhancement up to 50 times compared to an incident field. They are usually localized at the surface between the arms and may be both "electric" and "magnetic". While the former are of primary importance for SERS, the latter may be identified by trapping magnetic nanoparticles in their vicinity. The results are in very good agreement with the data where available and may be used as a type of a "shape spectroscopy" for the nanoparticles.

cond-mat.mes-hall↗