arXiv subjects

Tong Zhou

Publications and source records attributed to Tong Zhou.

At least 91 records · Page 5Linked to original sources

arXiv2023

A Survey on Multimodal Large Language Models for Autonomous Driving

With the emergence of Large Language Models (LLMs) and Vision Foundation Models (VFMs), multimodal AI systems benefiting from large models have the potential to equally perceive the real world, make decisions, and control tools as humans. In recent months, LLMs have shown widespread attention in autonomous driving and map systems. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors to apply in LLM driving systems. In this paper, we present a systematic investigation in this field. We first introduce the background of Multimodal Large Language Models (MLLMs), the multimodal models development using LLMs, and the history of autonomous driving. Then, we overview existing MLLM tools for driving, transportation, and map systems together with existing datasets and benchmarks. Moreover, we summarized the works in The 1st WACV Workshop on Large Language and Vision Models for Autonomous Driving (LLVM-AD), which is the first workshop of its kind regarding LLMs in autonomous driving. To further promote the development of this field, we also discuss several important problems regarding using MLLMs in autonomous driving systems that need to be solved by both academia and industry.

cs.AI↗

arXiv2023

Oasis: Data Curation and Assessment System for Pretraining of Large Language Models

Data is one of the most critical elements in building a large language model. However, existing systems either fail to customize a corpus curation pipeline or neglect to leverage comprehensive corpus assessment for iterative optimization of the curation. To this end, we present a pretraining corpus curation and assessment platform called Oasis -- a one-stop system for data quality improvement and quantification with user-friendly interactive interfaces. Specifically, the interactive modular rule filter module can devise customized rules according to explicit feedback. The debiased neural filter module builds the quality classification dataset in a negative-centric manner to remove the undesired bias. The adaptive document deduplication module could execute large-scale deduplication with limited memory resources. These three parts constitute the customized data curation module. And in the holistic data assessment module, a corpus can be assessed in local and global views, with three evaluation means including human, GPT-4, and heuristic metrics. We exhibit a complete process to use Oasis for the curation and assessment of pretraining data. In addition, an 800GB bilingual corpus curated by Oasis is publicly released.

cs.CL↗

arXiv2023

MirrorNet: A TEE-Friendly Framework for Secure On-device DNN Inference

Deep neural network (DNN) models have become prevalent in edge devices for real-time inference. However, they are vulnerable to model extraction attacks and require protection. Existing defense approaches either fail to fully safeguard model confidentiality or result in significant latency issues. To overcome these challenges, this paper presents MirrorNet, which leverages Trusted Execution Environment (TEE) to enable secure on-device DNN inference. It generates a TEE-friendly implementation for any given DNN model to protect the model confidentiality, while meeting the stringent computation and storage constraints of TEE. The framework consists of two key components: the backbone model (BackboneNet), which is stored in the normal world but achieves lower inference accuracy, and the Companion Partial Monitor (CPM), a lightweight mirrored branch stored in the secure world, preserving model confidentiality. During inference, the CPM monitors the intermediate results from the BackboneNet and rectifies the classification output to achieve higher accuracy. To enhance flexibility, MirrorNet incorporates two modules: the CPM Strategy Generator, which generates various protection strategies, and the Performance Emulator, which estimates the performance of each strategy and selects the most optimal one. Extensive experiments demonstrate the effectiveness of MirrorNet in providing security guarantees while maintaining low computation latency, making MirrorNet a practical and promising solution for secure on-device DNN inference. For the evaluation, MirrorNet can achieve a 18.6% accuracy gap between authenticated and illegal use, while only introducing 0.99% hardware overhead.

cs.CR↗

arXiv2023

LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios

Building agents based on tree-search planning capabilities with learned models has achieved remarkable success in classic decision-making problems, such as Go and Atari. However, it has been deemed challenging or even infeasible to extend Monte Carlo Tree Search (MCTS) based algorithms to diverse real-world applications, especially when these environments involve complex action spaces and significant simulation costs, or inherent stochasticity. In this work, we introduce LightZero, the first unified benchmark for deploying MCTS/MuZero in general sequential decision scenarios. Specificially, we summarize the most critical challenges in designing a general MCTS-style decision-making solver, then decompose the tightly-coupled algorithm and system design of tree-search RL methods into distinct sub-modules. By incorporating more appropriate exploration and optimization strategies, we can significantly enhance these sub-modules and construct powerful LightZero agents to tackle tasks across a wide range of domains, such as board games, Atari, MuJoCo, MiniGrid and GoBigger. Detailed benchmark results reveal the significant potential of such methods in building scalable and efficient decision intelligence. The code is available as part of OpenDILab at https://github.com/opendilab/LightZero.

cs.LG↗

arXiv2023

Towards High Efficient Long-horizon Planning with Expert-guided Motion-encoding Tree Search

Autonomous driving holds promise for increased safety, optimized traffic management, and a new level of convenience in transportation. While model-based reinforcement learning approaches such as MuZero enables long-term planning, the exponentially increase of the number of search nodes as the tree goes deeper significantly effect the searching efficiency. To deal with this problem, in this paper we proposed the expert-guided motion-encoding tree search (EMTS) algorithm. EMTS extends the MuZero algorithm by representing possible motions with a comprehensive motion primitives latent space and incorporating expert policies toimprove the searching efficiency. The comprehensive motion primitives latent space enables EMTS to sample arbitrary trajectories instead of raw action to reduce the depth of the search tree. And the incorporation of expert policies guided the search and training phases the EMTS algorithm to enable early convergence. In the experiment section, the EMTS algorithm is compared with other four algorithms in three challenging scenarios. The experiment result verifies the effectiveness and the searching efficiency of the proposed EMTS algorithm.

cs.RO↗

arXiv2023

Multi-granulariy Time-based Transformer for Knowledge Tracing

In this paper, we present a transformer architecture for predicting student performance on standardized tests. Specifically, we leverage students historical data, including their past test scores, study habits, and other relevant information, to create a personalized model for each student. We then use these models to predict their future performance on a given test. Applying this model to the RIIID dataset, we demonstrate that using multiple granularities for temporal features as the decoder input significantly improve model performance. Our results also show the effectiveness of our approach, with substantial improvements over the LightGBM method. Our work contributes to the growing field of AI in education, providing a scalable and accurate tool for predicting student outcomes.

cs.LG↗

arXiv2023

Nonparametric Identification and Estimation of Earnings Dynamics using a Hidden Markov Model: Evidence from the PSID

This paper presents a hidden Markov model designed to investigate the complex nature of earnings persistence. The proposed model assumes that the residuals of log-earnings consist of a persistent component and a transitory component, both following general Markov processes. Nonparametric identification is achieved through spectral decomposition of linear operators, and a modified stochastic EM algorithm is introduced for model estimation. Applying the framework to the Panel Study of Income Dynamics (PSID) dataset, we find that the earnings process displays nonlinear persistence, conditional skewness, and conditional kurtosis. Additionally, the transitory component is found to possess non-Gaussian properties, resulting in a significantly asymmetric distributional impact when high-earning households face negative shocks or low-earning households encounter positive shocks. Our empirical findings also reveal the presence of ARCH effects in earnings at horizons ranging from 2 to 8 years, further highlighting the complex dynamics of earnings persistence.

stat.AP↗

arXiv2023

AutoReP: Automatic ReLU Replacement for Fast Private Network Inference

The growth of the Machine-Learning-As-A-Service (MLaaS) market has highlighted clients' data privacy and security issues. Private inference (PI) techniques using cryptographic primitives offer a solution but often have high computation and communication costs, particularly with non-linear operators like ReLU. Many attempts to reduce ReLU operations exist, but they may need heuristic threshold selection or cause substantial accuracy loss. This work introduces AutoReP, a gradient-based approach to lessen non-linear operators and alleviate these issues. It automates the selection of ReLU and polynomial functions to speed up PI applications and introduces distribution-aware polynomial approximation (DaPa) to maintain model expressivity while accurately approximating ReLUs. Our experimental results demonstrate significant accuracy improvements of 6.12% (94.31%, 12.9K ReLU budget, CIFAR-10), 8.39% (74.92%, 12.9K ReLU budget, CIFAR-100), and 9.45% (63.69%, 55K ReLU budget, Tiny-ImageNet) over current state-of-the-art methods, e.g., SNL. Morever, AutoReP is applied to EfficientNet-B2 on ImageNet dataset, and achieved 75.55% accuracy with 176.1 times ReLU budget reduction.

cs.CR↗

arXiv2023

NNSplitter: An Active Defense Solution for DNN Model via Automated Weight Obfuscation

As a type of valuable intellectual property (IP), deep neural network (DNN) models have been protected by techniques like watermarking. However, such passive model protection cannot fully prevent model abuse. In this work, we propose an active model IP protection scheme, namely NNSplitter, which actively protects the model by splitting it into two parts: the obfuscated model that performs poorly due to weight obfuscation, and the model secrets consisting of the indexes and original values of the obfuscated weights, which can only be accessed by authorized users with the support of the trusted execution environment. Experimental results demonstrate the effectiveness of NNSplitter, e.g., by only modifying 275 out of over 11 million (i.e., 0.002%) weights, the accuracy of the obfuscated ResNet-18 model on CIFAR-10 can drop to 10%. Moreover, NNSplitter is stealthy and resilient against norm clipping and fine-tuning attacks, making it an appealing solution for DNN model protection. The code is available at: https://github.com/Tongzhou0101/NNSplitter.

cs.LG↗

arXiv2023

Liquid Metal Molecular Scissors

Molecules are the smallest unit in matters that can exist independently, relatively stable, and maintain physical and chemical activities. The atomic species, alignment commands, and chemical bonds are key factors to dominate their structures and properties. Here we disclosed a general chemistry effect that the liquid metals can directly cut off oxygen-containing groups in various molecular matters at room temperature, and then recombine the remaining groups to form functional materials including nano semiconductors. Based on this unique mechanism, we proposed a basic tool and named it as liquid metal scissors for molecular directional clipping and functional transformation. As proof-of-concept, we demonstrated the capabilities of eGaIn scissors made of Ga and In particles, and revealed that the Ga on the surface of eGaIn could directly snatch oxygen atoms from various targeted substances such as H2O, CO2 or CH3OH molecules to form gallium oxides. As illustration, after clipping, the remaining hydrogen atoms of H2O molecules recombined to form H2, while the remaining groups of CH3OH lead to H2, carbon quantum dots, and other related substances. If needed, more molecules can also be manipulated via such scissors. This finding refreshes the basic knowledge of chemistry and suggests easygoing ways for molecular weaving, which may break up the limitations and single features of molecular substances. It also opens up a universal route for innovating future molecular chemical engineering, life science, energy and environment, and biomedicine.

physics.chem-ph↗

arXiv2023

Epitaxial Kagome Thin Films as a Platform for Topological Flat Bands

Systems with flat bands are ideal for studying strongly correlated electronic states and related phenomena. Among them, kagome-structured metals such as CoSn have been recognized as promising candidates due to the proximity between the flat bands and the Fermi level. A key next step will be to realize epitaxial kagome thin films with flat bands to enable tuning of the flat bands across the Fermi level via electrostatic gating or strain. Here we report the band structures of epitaxial CoSn thin films grown directly on insulating substrates. Flat bands are observed using synchrotron-based angle-resolved photoemission spectroscopy (ARPES). The band structure is consistent with density functional theory (DFT) calculations, and the transport properties are quantitatively explained by the band structure and semiclassical transport theory. Our work paves the way to realize flat band-induced phenomena through fine-tuning of flat bands in kagome materials.

cond-mat.mtrl-sci↗

arXiv2023

Chromium Self-Traceable Length Standard: Investigating Geometry and Diffraction for Length Traceability Chain

Natural constant-based metrology methods offer an effective approach to achieving traceability in nanometric measurements. The Cr grating, fabricated by atom lithography and featuring a pitch of $d=212.7705\pm0.0049~{\rm nm}$ traceable to the Cr transition frequency $^{7}S_{3}$ $\rightarrow$ $^{7}P_{4}^{0}$, demonstrates potential as a self-traceable length standard in nano-length metrology by grating interferometer. This research aims to analyze and engineer the diffraction characteristics that enhance the Cr grating as a self-traceable length standard within the length traceability chain based on the Cr transition frequency. Accordingly, we investigate the geometric morphology and diffraction characteristics of the Cr grating, analyzes the influence of the grating's polarization-sensitive characteristics on the Littrow configuration grating interferometer, and establishes the criteria for Cr grating fabrication. Experimentally, we fabricate an expanded Cr grating by scanning atom lithography, characterize its diffraction performance, and conduct preliminary verification of length measurement in a self-traceable grating interferometer. This work adheres to the international trend of flattened metrology development, offering a valuable reference for advancing subsequent metrological technologies throughout the new traceability chain.

physics.ins-det↗

arXiv2023

Nanometer displacement measurement based on metrological self-mixing grating interferometer traceable to the pitch standard of one-dimension chromium self-traceable grating

Traceability of precision instrument and measuring method is the core issue in metrology science. In the field of nanometer length measurement, the laser interferometers are usually used to trace the measurement value to the laser wavelength, but the laser wavelength is sensitive to the environment disturbance. Chromium self-traceable grating is an ideal nanometer length reference grating with pitch traceability, fabricated by the atomic lithography technique. The new nanometer length traceability chain can be established based on the pitch traceability of chromium self-traceable grating, which is often used to calibrate the systematic error of the atomic force microscope. In this paper, the metrological self-mixing grating interferometer based on the chromium self-traceable grating (SMGI-Cr) is firstly established, whose interfere phase is traceable to the pitch of the chromium self-traceable grating directly and traceable to the chromium atomic transition frequency of energy level 7 S 3 to 7 P 4 indirectly. The nanometer displacement measurement is also achieved by the SMGI-Cr. The measurement error is no more than 0.2366%, compared to a commercial interferometer.

physics.app-ph↗

arXiv2023

Boosting Event Extraction with Denoised Structure-to-Text Augmentation

Event extraction aims to recognize pre-defined event triggers and arguments from texts, which suffer from the lack of high-quality annotations. In most NLP applications, involving a large scale of synthetic training data is a practical and effective approach to alleviate the problem of data scarcity. However, when applying to the task of event extraction, recent data augmentation methods often neglect the problem of grammatical incorrectness, structure misalignment, and semantic drifting, leading to unsatisfactory performances. In order to solve these problems, we propose a denoised structure-to-text augmentation framework for event extraction DAEE, which generates additional training data through the knowledge-based structure-to-text generation model and selects the effective subset from the generated data iteratively with a deep reinforcement learning agent. Experimental results on several datasets demonstrate that the proposed method generates more diverse text representations for event extraction and achieves comparable results with the state-of-the-art.

cs.CL↗

arXiv2023

Large and tunable magnetoresistance in van der Waals Ferromagnet/Semiconductor junctions

Magnetic tunnel junctions (MTJs) with conventional bulk ferromagnets separated by a nonmagnetic insulating layer are key building blocks in spintronics for magnetic sensors and memory. A radically different approach of using atomically-thin van der Waals (vdW) materials in MTJs is expected to boost their figure of merit, the tunneling magnetoresistance (TMR), while relaxing the lattice-matching requirements from the epitaxial growth and supporting high-quality integration of dissimilar materials with atomically-sharp interfaces. We report TMR up to 192% at 10 K in all-vdW Fe3GeTe2/GaSe/Fe3GeTe2 MTJs. Remarkably, instead of the usual insulating spacer, this large TMR is realized with a vdW semiconductor GaSe. Integration of two-dimensional ferromagnets in semiconductor-based vdW junctions offers gate-tunability, bias dependence, magnetic proximity effects, and spin-dependent optical-selection rules. We demonstrate that not just the magnitude, but also the TMR sign is tuned by the applied bias or the semiconductor thickness, enabling modulation of highly spin-polarized carriers in vdW semiconductors.

cond-mat.mtrl-sci↗

arXiv2022

THMA: Tencent HD Map AI System for Creating HD Map Annotations

Nowadays, autonomous vehicle technology is becoming more and more mature. Critical to progress and safety, high-definition (HD) maps, a type of centimeter-level map collected using a laser sensor, provide accurate descriptions of the surrounding environment. The key challenge of HD map production is efficient, high-quality collection and annotation of large-volume datasets. Due to the demand for high quality, HD map production requires significant manual human effort to create annotations, a very time-consuming and costly process for the map industry. In order to reduce manual annotation burdens, many artificial intelligence (AI) algorithms have been developed to pre-label the HD maps. However, there still exists a large gap between AI algorithms and the traditional manual HD map production pipelines in accuracy and robustness. Furthermore, it is also very resource-costly to build large-scale annotated datasets and advanced machine learning algorithms for AI-based HD map automatic labeling systems. In this paper, we introduce the Tencent HD Map AI (THMA) system, an innovative end-to-end, AI-based, active learning HD map labeling system capable of producing and labeling HD maps with a scale of hundreds of thousands of kilometers. In THMA, we train AI models directly from massive HD map datasets via supervised, self-supervised, and weakly supervised learning to achieve high accuracy and efficiency required by downstream users. THMA has been deployed by the Tencent Map team to provide services to downstream companies and users, serving over 1,000 labeling workers and producing more than 30,000 kilometers of HD map data per day at most. More than 90 percent of the HD map data in Tencent Map is labeled automatically by THMA, accelerating the traditional HD map labeling process by more than ten times.

cs.CV↗

arXiv2022

Accelerating Reinforcement Learning for Autonomous Driving using Task-Agnostic and Ego-Centric Motion Skills

Efficient and effective exploration in continuous space is a central problem in applying reinforcement learning (RL) to autonomous driving. Skills learned from expert demonstrations or designed for specific tasks can benefit the exploration, but they are usually costly-collected, unbalanced/sub-optimal, or failing to transfer to diverse tasks. However, human drivers can adapt to varied driving tasks without demonstrations by taking efficient and structural explorations in the entire skill space rather than a limited space with task-specific skills. Inspired by the above fact, we propose an RL algorithm exploring all feasible motion skills instead of a limited set of task-specific and object-centric skills. Without demonstrations, our method can still perform well in diverse tasks. First, we build a task-agnostic and ego-centric (TaEc) motion skill library in a pure motion perspective, which is diverse enough to be reusable in different complex tasks. The motion skills are then encoded into a low-dimension latent skill space, in which RL can do exploration efficiently. Validations in various challenging driving scenarios demonstrate that our proposed method, TaEc-RL, outperforms its counterparts significantly in learning efficiency and task performance.

cs.RO↗

arXiv2022

ObfuNAS: A Neural Architecture Search-based DNN Obfuscation Approach

Malicious architecture extraction has been emerging as a crucial concern for deep neural network (DNN) security. As a defense, architecture obfuscation is proposed to remap the victim DNN to a different architecture. Nonetheless, we observe that, with only extracting an obfuscated DNN architecture, the adversary can still retrain a substitute model with high performance (e.g., accuracy), rendering the obfuscation techniques ineffective. To mitigate this under-explored vulnerability, we propose ObfuNAS, which converts the DNN architecture obfuscation into a neural architecture search (NAS) problem. Using a combination of function-preserving obfuscation strategies, ObfuNAS ensures that the obfuscated DNN architecture can only achieve lower accuracy than the victim. We validate the performance of ObfuNAS with open-source architecture datasets like NAS-Bench-101 and NAS-Bench-301. The experimental results demonstrate that ObfuNAS can successfully find the optimal mask for a victim model within a given FLOPs constraint, leading up to 2.6% inference accuracy degradation for attackers with only 0.14x FLOPs overhead. The code is available at: https://github.com/Tongzhou0101/ObfuNAS.

cs.CR↗