arXiv subjects

Wei Li

Publications and source records attributed to Wei Li.

At least 613 records · Page 34Linked to original sources

arXiv2024

The Binary Quantized Neural Network for Dense Prediction via Specially Designed Upsampling and Attention

Deep learning-based information processing consumes long time and requires huge computing resources, especially for dense prediction tasks which require an output for each pixel, like semantic segmentation and salient object detection. There are mainly two challenges for quantization of dense prediction tasks. Firstly, directly applying the upsampling operation that dense prediction tasks require is extremely crude and causes unacceptable accuracy reduction. Secondly, the complex structure of dense prediction networks means it is difficult to maintain a fast speed as well as a high accuracy when performing quantization. In this paper, we propose an effective upsampling method and an efficient attention computation strategy to transfer the success of the binary neural networks (BNN) from single prediction tasks to dense prediction tasks. Firstly, we design a simple and robust multi-branch parallel upsampling structure to achieve the high accuracy. Then we further optimize the attention method which plays an important role in segmentation but has huge computation complexity. Our attention method can reduce the computational complexity by a factor of one hundred times but retain the original effect. Experiments on Cityscapes, KITTI road, and ECSSD fully show the effectiveness of our work.

cs.LG↗

arXiv2024

Detection-Correction Structure via General Language Model for Grammatical Error Correction

Grammatical error correction (GEC) is a task dedicated to rectifying texts with minimal edits, which can be decoupled into two components: detection and correction. However, previous works have predominantly focused on direct correction, with no prior efforts to integrate both into a single model. Moreover, the exploration of the detection-correction paradigm by large language models (LLMs) remains underdeveloped. This paper introduces an integrated detection-correction structure, named DeCoGLM, based on the General Language Model (GLM). The detection phase employs a fault-tolerant detection template, while the correction phase leverages autoregressive mask infilling for localized error correction. Through the strategic organization of input tokens and modification of attention masks, we facilitate multi-task learning within a single model. Our model demonstrates competitive performance against the state-of-the-art models on English and Chinese GEC datasets. Further experiments present the effectiveness of the detection-correction structure in LLMs, suggesting a promising direction for GEC.

cs.CL↗

arXiv2024

Lifelong Learning and Selective Forgetting via Contrastive Strategy

Lifelong learning aims to train a model with good performance for new tasks while retaining the capacity of previous tasks. However, some practical scenarios require the system to forget undesirable knowledge due to privacy issues, which is called selective forgetting. The joint task of the two is dubbed Learning with Selective Forgetting (LSF). In this paper, we propose a new framework based on contrastive strategy for LSF. Specifically, for the preserved classes (tasks), we make features extracted from different samples within a same class compacted. And for the deleted classes, we make the features from different samples of a same class dispersed and irregular, i.e., the network does not have any regular response to samples from a specific deleted class as if the network has no training at all. Through maintaining or disturbing the feature distribution, the forgetting and memory of different classes can be or independent of each other. Experiments are conducted on four benchmark datasets, and our method acieves new state-of-the-art.

cs.AI↗

arXiv2024

Combining Radiomics and Machine Learning Approaches for Objective ASD Diagnosis: Verifying White Matter Associations with ASD

Autism Spectrum Disorder is a condition characterized by a typical brain development leading to impairments in social skills, communication abilities, repetitive behaviors, and sensory processing. There have been many studies combining brain MRI images with machine learning algorithms to achieve objective diagnosis of autism, but the correlation between white matter and autism has not been fully utilized. To address this gap, we develop a computer-aided diagnostic model focusing on white matter regions in brain MRI by employing radiomics and machine learning methods. This study introduced a MultiUNet model for segmenting white matter, leveraging the UNet architecture and utilizing manually segmented MRI images as the training data. Subsequently, we extracted white matter features using the Pyradiomics toolkit and applied different machine learning models such as Support Vector Machine, Random Forest, Logistic Regression, and K-Nearest Neighbors to predict autism. The prediction sets all exceeded 80% accuracy. Additionally, we employed Convolutional Neural Network to analyze segmented white matter images, achieving a prediction accuracy of 86.84%. Notably, Support Vector Machine demonstrated the highest prediction accuracy at 89.47%. These findings not only underscore the efficacy of the models but also establish a link between white matter abnormalities and autism. Our study contributes to a comprehensive evaluation of various diagnostic models for autism and introduces a computer-aided diagnostic algorithm for early and objective autism diagnosis based on MRI white matter regions.

eess.IV↗

arXiv2024

Collaboration of Teachers for Semi-supervised Object Detection

Recent semi-supervised object detection (SSOD) has achieved remarkable progress by leveraging unlabeled data for training. Mainstream SSOD methods rely on Consistency Regularization methods and Exponential Moving Average (EMA), which form a cyclic data flow. However, the EMA updating training approach leads to weight coupling between the teacher and student models. This coupling in a cyclic data flow results in a decrease in the utilization of unlabeled data information and the confirmation bias on low-quality or erroneous pseudo-labels. To address these issues, we propose the Collaboration of Teachers Framework (CTF), which consists of multiple pairs of teacher and student models for training. In the learning process of CTF, the Data Performance Consistency Optimization module (DPCO) informs the best pair of teacher models possessing the optimal pseudo-labels during the past training process, and these most reliable pseudo-labels generated by the best performing teacher would guide the other student models. As a consequence, this framework greatly improves the utilization of unlabeled data and prevents the positive feedback cycle of unreliable pseudo-labels. The CTF achieves outstanding results on numerous SSOD datasets, including a 0.71% mAP improvement on the 10% annotated COCO dataset and a 0.89% mAP improvement on the VOC dataset compared to LabelMatch and converges significantly faster. Moreover, the CTF is plug-and-play and can be integrated with other mainstream SSOD methods.

cs.CV↗

arXiv2024

Kinetic energy and speed powers $v^n$ of a heavy quark inside $S$ wave and $P$ wave heavy-light mesons

Based on the instantaneous Bethe-Salpeter equation method, we calculate the average values $\overline{|\vec{q}|^n}\equiv q^n$ and speed powers $\overline{|\vec{v}|^n} \equiv v^n$ ($n=1,2,3,4$) of a heavy quark inside $S$ wave and $P$ wave heavy-light mesons, where $\vec{q}$ and $\vec{v}$ are the three dimensional momentum and velocity of the heavy quark, respectively. We obtain the kinetic energy $μ^2_{_π}=0.455$ GeV$^2$ for the $B$ meson, which is consistent with the experimental result $0.464\pm 0.076$ GeV$^2$. For the $B_{s}$, $D$ and $D_{s}$ mesons, the $μ^2_{_π}$ are $0.530$ GeV$^2$, $0.317$ GeV$^2$ and $0.369$ GeV$^2$, respectively. And $v^2=0.0185$, $0.0215$, $0.121$, and $0.140$ for $B$, $B_{s}$, $D$, and $D_{s}$. We obtain some relationships, for example, $q^n_{_{0^-}}(mS)\approx q^n_{_{1^-}}(mS)$, $q^n_{_{0^+}}(mP)\approx q^n_{_{1^{+'}}}(mP^{'})> q^n_{_{1^+}}(mP)\approx q^n_{_{2^+}}(mP)$, and $q^n(mS)< q^n(mP)$ ($m=1,2,3$), etc.

hep-ph↗

arXiv2024

Picking watermarks from noise (PWFN): an improved robust watermarking model against intensive distortions

Digital watermarking is the process of embedding secret information by altering images in an undetectable way to the human eye. To increase the robustness of the model, many deep learning-based watermarking methods use the encoder-noise-decoder architecture by adding different noises to the noise layer. The decoder then extracts the watermarked information from the distorted image. However, this method can only resist weak noise attacks. To improve the robustness of the decoder against stronger noise, this paper proposes to introduce a denoise module between the noise layer and the decoder. The module aims to reduce noise and recover some of the information lost caused by distortion. Additionally, the paper introduces the SE module to fuse the watermarking information pixel-wise and channel dimensions-wise, improving the encoder's efficiency. Experimental results show that our proposed method is comparable to existing models and outperforms state-of-the-art under different noise intensities. In addition, ablation experiments show the superiority of our proposed module.

cs.MM↗

arXiv2024

Evidence for distortion-induced local electric polarization in $α$-RuCl$_3$

The spin-orbit assisted Mott insulator $α$-RuCl$_3$ is a prime candidate for material realization of the Kitaev quantum spin liquid. While little attention has been paid to charge degrees of freedom, charge effects, such as electric polarization, may arise in this system. Here, we report distortion-induced local electric polarization in $α$-RuCl$_3$ as evidenced by single-crystal X-ray diffraction, second harmonic generation (SHG) and dielectric measurements. The SHG signal appears at room temperature and develops substantially in the Kitaev paramagnetic state when short-range spin correlations come into play. Despite sizable pyroelectric currents in the Kitaev paramagnetic state, the absence of hysteresis in electric field-dependent polariza-tion (P-E) points to the short-range nature of electric polarization. This localized electric polarization is likely the result of distortion-induced charge dimerization, achieved through virtual hopping-induced charge redistribution. In addition, the electric polarization is boosted by short-range spin correlations via spin-phonon coupling in the Kitaev paramagnetic state. Our results emphasize the importance of charge degrees of freedom in $α$-RuCl$_3$, which establish a novel platform to investi-gate charge effects in Kitaev materials.

cond-mat.str-el↗

arXiv2024

Preliminary Design of a General Electronics Platform for Accelerator Facilities

Many accelerators require considerable electronic systems for tests, verification, and operation. In Shenzhen Superconducting Soft X-ray Free Electron Laser (S3FEL), to meet the early tests and verification of various systems, save development expenses, and improve the reusability of hardware, firmware, and software systems, we have considered the needs of each system and preliminarily designed a general electronics platform based on MicroTCA.4. The Advanced Mezzanine Card (AMC) will place an FPGA Mezzanine Card (FMC) that supports 500 MSPS to 2 GSPS ADC/DAC. We will design two FMC cards on the Rear Transition Module (RTM), which can be used for analog signal conditioning and waveform digitization by 10 MSPS to 250 MSPS ADC/DAC or motor control. The commercial MCH, CPU, power module, and MTCA crate are deployed. This platform can also be applied to other accelerator facilities.

physics.ins-det↗

arXiv2024

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks

Due to its simplicity and efficiency, the first-order gradient method has been extensively employed in training neural networks. Although the optimization problem of the neural network is non-convex, recent research has proved that the first-order method is capable of attaining a global minimum during training over-parameterized neural networks, where the number of parameters is significantly larger than that of training instances. Momentum methods, including the heavy ball (HB) method and Nesterov's accelerated gradient (NAG) method, are the workhorse of first-order gradient methods owning to their accelerated convergence. In practice, NAG often exhibits superior performance than HB. However, current theoretical works fail to distinguish their convergence difference in training neural networks. To fill this gap, we consider the training problem of the two-layer ReLU neural network under over-parameterization and random initialization. Leveraging high-resolution dynamical systems and neural tangent kernel (NTK) theory, our result not only establishes tighter upper bounds of the convergence rate for both HB and NAG, but also provides the first theoretical guarantee for the acceleration of NAG over HB in training neural networks. Finally, we validate our theoretical results on three benchmark datasets.

cs.LG↗

arXiv2024

Interface Modes in Honeycomb Topological Photonic Structures with Broken Reflection Symmetry

In this work, we present a mathematical theory for Dirac points and interface modes in honeycomb topological photonic structures consisting of impenetrable obstacles. Starting from a honeycomb lattice of obstacles attaining $120^\circ$-rotation symmetry and horizontal reflection symmetry, we apply the boundary integral equation method to show the existence of Dirac points for the first two bands at the vertices of the Brillouin zone. We then study interface modes in a joint honeycomb photonic structure, which consists of two periodic lattices obtained by perturbing the honeycomb one with Dirac points differently. The perturbations break the reflection symmetry of the system, as a result, they annihilate the Dirac points and generate two structures with different topological phases, which mimics the quantum valley Hall effect in topological insulators. We investigate the interface modes that decay exponentially away from the interface of the joint structure in several configurations with different interface geometries, including the zigzag interface, the armchair interface, and the rational interfaces. Using the layer potential technique and asymptotic analysis, we first characterize the band-gap opening for the two perturbed periodic structures and derive the asymptotic expansions of the Bloch modes near the band gap surfaces. By formulating the eigenvalue problem for each joint honeycomb structure using boundary integral equations over the interface and analyzing the characteristic values of the associated boundary integral operators, we prove the existence of interface modes when the perturbation is small.

math-ph↗

arXiv2024

ESP: Extro-Spective Prediction for Long-term Behavior Reasoning in Emergency Scenarios

Emergent-scene safety is the key milestone for fully autonomous driving, and reliable on-time prediction is essential to maintain safety in emergency scenarios. However, these emergency scenarios are long-tailed and hard to collect, which restricts the system from getting reliable predictions. In this paper, we build a new dataset, which aims at the long-term prediction with the inconspicuous state variation in history for the emergency event, named the Extro-Spective Prediction (ESP) problem. Based on the proposed dataset, a flexible feature encoder for ESP is introduced to various prediction methods as a seamless plug-in, and its consistent performance improvement underscores its efficacy. Furthermore, a new metric named clamped temporal error (CTE) is proposed to give a more comprehensive evaluation of prediction performance, especially in time-sensitive emergency events of subseconds. Interestingly, as our ESP features can be described in human-readable language naturally, the application of integrating into ChatGPT also shows huge potential. The ESP-dataset and all benchmarks are released at https://dingrui-wang.github.io/ESP-Dataset/.

cs.CV↗

arXiv2024

LIPT: Latency-aware Image Processing Transformer

Transformer is leading a trend in the field of image processing. Despite the great success that existing lightweight image processing transformers have achieved, they are tailored to FLOPs or parameters reduction, rather than practical inference acceleration. In this paper, we present a latency-aware image processing transformer, termed LIPT. We devise the low-latency proportion LIPT block that substitutes memory-intensive operators with the combination of self-attention and convolutions to achieve practical speedup. Specifically, we propose a novel non-volatile sparse masking self-attention (NVSM-SA) that utilizes a pre-computing sparse mask to capture contextual information from a larger window with no extra computation overload. Besides, a high-frequency reparameterization module (HRM) is proposed to make LIPT block reparameterization friendly, which improves the model's detail reconstruction capability. Extensive experiments on multiple image processing tasks (e.g., image super-resolution (SR), JPEG artifact reduction, and image denoising) demonstrate the superiority of LIPT on both latency and PSNR. LIPT achieves real-time GPU inference with state-of-the-art performance on multiple image SR benchmarks.

cs.CV↗

arXiv2024

How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements: (1) Strong Vision Encoder: we explored a continuous learning strategy for the large-scale vision foundation model -- InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic High-Resolution: we divide images into tiles ranging from 1 to 40 of 448$\times$448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) High-Quality Bilingual Dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in OCR- and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks. Code has been released at https://github.com/OpenGVLab/InternVL.

cs.CV↗

arXiv2024

FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models

In the burgeoning field of large language models (LLMs), the assessment of fundamental knowledge remains a critical challenge, particularly for models tailored to Chinese language and culture. This paper introduces FoundaBench, a pioneering benchmark designed to rigorously evaluate the fundamental knowledge capabilities of Chinese LLMs. FoundaBench encompasses a diverse array of 3354 multiple-choice questions across common sense and K-12 educational subjects, meticulously curated to reflect the breadth and depth of everyday and academic knowledge. We present an extensive evaluation of 12 state-of-the-art LLMs using FoundaBench, employing both traditional assessment methods and our CircularEval protocol to mitigate potential biases in model responses. Our results highlight the superior performance of models pre-trained on Chinese corpora, and reveal a significant disparity between models' reasoning and memory recall capabilities. The insights gleaned from FoundaBench evaluations set a new standard for understanding the fundamental knowledge of LLMs, providing a robust framework for future advancements in the field.

cs.CL↗

arXiv2024

Orbital-selective Superconductivity in the Pressurized Bilayer Nickelate La$_3$Ni$_2$O$_7$: An Infinite Projected Entangled-Pair State Study

The newly discovered high-$T_c$ nickelate superconductor La$_3$Ni$_2$O$_7$ has generated significant research interest. To uncover the pairing mechanism, it is essential to investigate the intriguing interplay between the two $e_g$, i.e., $d_{x^2-y^2}$ and $d_{z^2}$ orbitals. Here we perform an infinite projected entangled-pair state (iPEPS) study of the bilayer $t$-$J$ model, directly in the thermodynamic limit and with orbitally selective parameters for $d_{x^2-y^2}$ and $d_{z^2}$ orbitals, respectively. The $d_{x^2-y^2}$ electrons exhibit significant intralayer hopping $t_\parallel$ and spin couplings $J_\parallel$, with interlayer $J_\perp$ passed from the $d_{z^2}$ electrons. However, the interlayer $t_\perp$ is negligible in this case. In contrast, the $d_{z^2}$ orbital demonstrates strong interlayer $t_\perp$ and $J_\perp$, while the inherent intralayer $t_\parallel$ and $J_\parallel$ are small. Based on the iPEPS results, we find clear orbital-selective behaviors in La$_3$Ni$_2$O$_7$. The $d_{x^2-y^2}$ orbitals exhibit robust superconductive (SC) order driven by the interlayer coupling $J_\perp$; while the $d_{z^2}$ band shows relatively weak SC order as a result of small $t_\parallel$ (lack of coherence) but large $t_\perp$ (strong Pauli blocking). Furthermore, by substituting rare-earth element Pm or Sm with La, we find an enhanced SC order, which opens up a promising avenue for discovering nickelate superconductors with even higher $T_c$.

cond-mat.str-el↗

arXiv2024

Data Upcycling Knowledge Distillation for Image Super-Resolution

Knowledge distillation (KD) compresses deep neural networks by transferring task-related knowledge from cumbersome pre-trained teacher models to compact student models. However, current KD methods for super-resolution (SR) networks overlook the nature of SR task that the outputs of the teacher model are noisy approximations to the ground-truth distribution of high-quality images (GT), which shades the teacher model's knowledge to result in limited KD effects. To utilize the teacher model beyond the GT upper-bound, we present the Data Upcycling Knowledge Distillation (DUKD), to transfer the teacher model's knowledge to the student model through the upcycled in-domain data derived from training data. Besides, we impose label consistency regularization to KD for SR by the paired invertible augmentations to improve the student model's performance and robustness. Comprehensive experiments demonstrate that the DUKD method significantly outperforms previous arts on several SR tasks.

cs.CV↗

arXiv2024

BezierFormer: A Unified Architecture for 2D and 3D Lane Detection

Lane detection has made significant progress in recent years, but there is not a unified architecture for its two sub-tasks: 2D lane detection and 3D lane detection. To fill this gap, we introduce BézierFormer, a unified 2D and 3D lane detection architecture based on Bézier curve lane representation. BézierFormer formulate queries as Bézier control points and incorporate a novel Bézier curve attention mechanism. This attention mechanism enables comprehensive and accurate feature extraction for slender lane curves via sampling and fusing multiple reference points on each curve. In addition, we propose a novel Chamfer IoU-based loss which is more suitable for the Bézier control points regression. The state-of-the-art performance of BézierFormer on widely-used 2D and 3D lane detection benchmarks verifies its effectiveness and suggests the worthiness of further exploration.

cs.CV↗