arXiv subjects

Junyan Li

Publications and source records attributed to Junyan Li.

29 records · Page 2Linked to original sources

arXiv2025

ISAM-MTL: Cross-subject multi-task learning model with identifiable spikes and associative memory networks

Cross-subject variability in EEG degrades performance of current deep learning models, limiting the development of brain-computer interface (BCI). This paper proposes ISAM-MTL, which is a multi-task learning (MTL) EEG classification model based on identifiable spiking (IS) representations and associative memory (AM) networks. The proposed model treats EEG classification of each subject as an independent task and leverages cross-subject data training to facilitate feature sharing across subjects. ISAM-MTL consists of a spiking feature extractor that captures shared features across subjects and a subject-specific bidirectional associative memory network that is trained by Hebbian learning for efficient and fast within-subject EEG classification. ISAM-MTL integrates learned spiking neural representations with bidirectional associative memory for cross-subject EEG classification. The model employs label-guided variational inference to construct identifiable spike representations, enhancing classification accuracy. Experimental results on two BCI Competition datasets demonstrate that ISAM-MTL improves the average accuracy of cross-subject EEG classification while reducing performance variability among subjects. The model further exhibits the characteristics of few-shot learning and identifiable neural activity beneath EEG, enabling rapid and interpretable calibration for BCI systems.

cs.NE↗

arXiv2024

Establishing a New Benchmark in Quantum Computational Advantage with 105-qubit Zuchongzhi 3.0 Processor

In the relentless pursuit of quantum computational advantage, we present a significant advancement with the development of Zuchongzhi 3.0. This superconducting quantum computer prototype, comprising 105 qubits, achieves high operational fidelities, with single-qubit gates, two-qubit gates, and readout fidelity at 99.90%, 99.62% and 99.18%, respectively. Our experiments with an 83-qubit, 32-cycle random circuit sampling on Zuchongzhi 3.0 highlight its superior performance, achieving one million samples in just a few hundred seconds. This task is estimated to be infeasible on the most powerful classical supercomputers, Frontier, which would require approximately $6.4\times 10^9$ years to replicate the task. This leap in processing power places the classical simulation cost six orders of magnitude beyond Google's SYC-67 and SYC-70 experiments [Nature 634, 328(2024)], firmly establishing a new benchmark in quantum computational advantage. Our work not only advances the frontiers of quantum computing but also lays the groundwork for a new era where quantum processors play an essential role in tackling sophisticated real-world challenges.

quant-ph↗

arXiv2024

Quantum-limited superresolution of two arbitrary incoherent point sources: beating the resurgence of Rayleigh's curse

Abstract Superresolution has been demonstrated to overcome the limitation of the Rayleigh's criterion and achieve significant improvement of the precision in resolving the separation of two incoherent optical point sources. However, in recent years, it was found that if the photon numbers of the two incoherent optical sources are unknown, the precision of superresolution vanishes when the two photon numbers are actually different. In this work, we first analyze the estimation precision of the separation between two incoherent optical sources with the same point-spread functions in detail, and show that when the photon numbers of the two optical sources are different but sufficiently close, the superresolution can still realized but with different precisions. We find the condition on how close the photon numbers of two optical sources need to be to realize the superresolution, and derive the precision of superresolution in different regimes of the photon number difference. We further consider the superresolution for two incoherent optical sources with different point-spread functions, and show that the competition between the difference of photon numbers, the difference of the two point-spread functions and the separation of the two optical sources determines the precision of superresolution. The results exhibit precision limits distinct from the case of two point sources with identical point-spread functions and equal photon numbers, and extend the realizable regimes of the quantum superresolution technique. The results are finally illustrated by Gaussian point-spread functions.

quant-ph↗

arXiv2024

AM-MTEEG: Multi-task EEG classification based on impulsive associative memory

Electroencephalogram-based brain-computer interface (BCI) has potential applications in various fields, but their development is hindered by limited data and significant cross-individual variability. Inspired by the principles of learning and memory in the human hippocampus, we propose a multi-task (MT) classification model, called AM-MTEEG, which combines learning-based impulsive neural representations with bidirectional associative memory (AM) for cross-individual BCI classification tasks. The model treats the EEG classification of each individual as an independent task and facilitates feature sharing across individuals. Our model consists of an impulsive neural population coupled with a convolutional encoder-decoder to extract shared features and a bidirectional associative memory matrix to map features to class. Experimental results in two BCI competition datasets show that our model improves average accuracy compared to state-of-the-art models and reduces performance variance across individuals, and the waveforms reconstructed by the bidirectional associative memory provide interpretability for the model's classification results. The neuronal firing patterns in our model are highly coordinated, similarly to the neural coding of hippocampal neurons, indicating that our model has biological similarities.

cs.NE↗

arXiv2024

FlexAttention for Efficient High-Resolution Vision-Language Models

Current high-resolution vision-language models encode images as high-resolution image tokens and exhaustively take all these tokens to compute attention, which significantly increases the computational cost. To address this problem, we propose FlexAttention, a flexible attention mechanism for efficient high-resolution vision-language models. Specifically, a high-resolution image is encoded both as high-resolution tokens and low-resolution tokens, where only the low-resolution tokens and a few selected high-resolution tokens are utilized to calculate the attention map, which greatly shrinks the computational cost. The high-resolution tokens are selected via a high-resolution selection module which could retrieve tokens of relevant regions based on an input attention map. The selected high-resolution tokens are then concatenated to the low-resolution tokens and text tokens, and input to a hierarchical self-attention layer which produces an attention map that could be used for the next-step high-resolution token selection. The hierarchical self-attention process and high-resolution token selection process are performed iteratively for each attention layer. Experiments on multimodal benchmarks prove that our FlexAttention outperforms existing high-resolution VLMs (e.g., relatively ~9% in V* Bench, ~7% in TextVQA), while also significantly reducing the computational cost by nearly 40%.

cs.CV↗

arXiv2024

EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction

High-resolution dense prediction enables many appealing real-world applications, such as computational photography, autonomous driving, etc. However, the vast computational cost makes deploying state-of-the-art high-resolution dense prediction models on hardware devices difficult. This work presents EfficientViT, a new family of high-resolution vision models with novel multi-scale linear attention. Unlike prior high-resolution dense prediction models that rely on heavy softmax attention, hardware-inefficient large-kernel convolution, or complicated topology structure to obtain good performances, our multi-scale linear attention achieves the global receptive field and multi-scale learning (two desirable features for high-resolution dense prediction) with only lightweight and hardware-efficient operations. As such, EfficientViT delivers remarkable performance gains over previous state-of-the-art models with significant speedup on diverse hardware platforms, including mobile CPU, edge GPU, and cloud GPU. Without performance loss on Cityscapes, our EfficientViT provides up to 13.9$\times$ and 6.2$\times$ GPU latency reduction over SegFormer and SegNeXt, respectively. For super-resolution, EfficientViT delivers up to 6.4x speedup over Restormer while providing 0.11dB gain in PSNR. For Segment Anything, EfficientViT delivers 48.9x higher throughput on A100 GPU while achieving slightly better zero-shot instance segmentation performance on COCO.

cs.CV↗

arXiv2024

MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World

Human beings possess the capability to multiply a melange of multisensory cues while actively exploring and interacting with the 3D world. Current multi-modal large language models, however, passively absorb sensory data as inputs, lacking the capacity to actively interact with the objects in the 3D environment and dynamically collect their multisensory information. To usher in the study of this area, we propose MultiPLY, a multisensory embodied large language model that could incorporate multisensory interactive data, including visual, audio, tactile, and thermal information into large language models, thereby establishing the correlation among words, actions, and percepts. To this end, we first collect Multisensory Universe, a large-scale multisensory interaction dataset comprising 500k data by deploying an LLM-powered embodied agent to engage with the 3D environment. To perform instruction tuning with pre-trained LLM on such generated data, we first encode the 3D scene as abstracted object-centric representations and then introduce action tokens denoting that the embodied agent takes certain actions within the environment, as well as state tokens that represent the multisensory state observations of the agent at each time step. In the inference time, MultiPLY could generate action tokens, instructing the agent to take the action in the environment and obtain the next multisensory state observation. The observation is then appended back to the LLM via state tokens to generate subsequent text or action tokens. We demonstrate that MultiPLY outperforms baselines by a large margin through a diverse set of embodied tasks involving object retrieval, tool use, multisensory captioning, and task decomposition.

cs.CV↗

arXiv2023

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

A remarkable ability of human beings resides in compositional reasoning, i.e., the capacity to make "infinite use of finite means". However, current large vision-language foundation models (VLMs) fall short of such compositional abilities due to their "bag-of-words" behaviors and inability to construct words that correctly represent visual entities and the relations among the entities. To this end, we propose CoVLM, which can guide the LLM to explicitly compose visual entities and relationships among the text and dynamically communicate with the vision encoder and detection network to achieve vision-language communicative decoding. Specifically, we first devise a set of novel communication tokens for the LLM, for dynamic communication between the visual detection system and the language system. A communication token is generated by the LLM following a visual entity or a relation, to inform the detection network to propose regions that are relevant to the sentence generated so far. The proposed regions-of-interests (ROIs) are then fed back into the LLM for better language generation contingent on the relevant regions. The LLM is thus able to compose the visual entities and relationships through the communication tokens. The vision-to-language and language-to-vision communication are iteratively performed until the entire sentence is generated. Our framework seamlessly bridges the gap between visual perception and LLMs and outperforms previous VLMs by a large margin on compositional reasoning benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on ARO top-1 accuracy). We also achieve state-of-the-art performances on traditional vision-language tasks such as referring expression comprehension and visual question answering.

cs.CV↗

arXiv2023

Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inference

Deploying pre-trained transformer models like BERT on downstream tasks in resource-constrained scenarios is challenging due to their high inference cost, which grows rapidly with input sequence length. In this work, we propose a constraint-aware and ranking-distilled token pruning method ToP, which selectively removes unnecessary tokens as input sequence passes through layers, allowing the model to improve online inference speed while preserving accuracy. ToP overcomes the limitation of inaccurate token importance ranking in the conventional self-attention mechanism through a ranking-distilled token distillation technique, which distills effective token rankings from the final layer of unpruned models to early layers of pruned models. Then, ToP introduces a coarse-to-fine pruning approach that automatically selects the optimal subset of transformer layers and optimizes token pruning decisions within these layers through improved $L_0$ regularization. Extensive experiments on GLUE benchmark and SQuAD tasks demonstrate that ToP outperforms state-of-the-art token pruning and model compression methods with improved accuracy and speedups. ToP reduces the average FLOPs of BERT by 8.1x while achieving competitive accuracy on GLUE, and provides a real latency speedup of up to 7.4x on an Intel CPU.

cs.CL↗

arXiv2023

Servo: Increasing the Scalability of Modifiable Virtual Environments Using Serverless Computing -- Extended Technical Report

Online games with modifiable virtual environments (MVEs) have become highly popular over the past decade. Among them, Minecraft -- supporting hundreds of millions of users -- is the best-selling game of all time, and is increasingly offered as a service. Although Minecraft is architected as a distributed system, in production it achieves this scale by partitioning small groups of players over isolated game instances. From the approaches that can help other kinds of virtual worlds scale, none is designed to scale MVEs, which pose a unique challenge -- a mix between the count and complexity of active in-game constructs, player-created in-game programs, and strict quality of service. Serverless computing emerged recently and focuses, among others, on service scalability. Thus, addressing this challenge, in this work we explore using serverless computing to improve MVE scalability. To this end, we design, prototype, and evaluate experimentally Servo, a serverless backend architecture for MVEs. We implement Servo as a prototype and evaluate it using real-world experiments on two commercial serverless platforms, of Amazon Web Services (AWS) and Microsoft Azure. Results offer strong support that our serverless MVE can significantly increase the number of supported players per instance without performance degradation, in our key experiment by 40 to 140 players per instance, which is a significant improvement over state-of-the-art commercial and open-source alternatives. We release Servo as open-source, on Github: https://github.com/atlarge-research/opencraft

cs.DC↗

arXiv2021

Random lasers with tunable angular spectra for high contrast imaging

Random lasers with low spatial coherence have important potential applications in high quality imaging and displaying. Here, a random laser with tunable angular spectra is proposed and fabricated through directly coupling an asymmetric microcavity with a commercial optical fiber. The designed random laser demonstrates pumping direction-independence property in working threshold, and good storage stability.More importantly, the angular spectrum can be adjusted by pumping different microcavities on the fiber, the output random lasing with ring-shape in momentum space are used as illumination source for biological imaging. An excellent image with speckle-free and higher contrast is achieved due to the low spatial frequency-free of the illumination source even relative to that of the common white lighting. The results indicate that the designed random laser has great application values in the fields of high-resolution biological imaging, integration optics and smart sensing.

physics.optics↗