arXiv subjects

Wei Lin

Publications and source records attributed to Wei Lin.

At least 181 records · Page 10Linked to original sources

arXiv2024

Governing equation discovery of a complex system from snapshots

Complex systems in physics, chemistry, and biology that evolve over time with inherent randomness are typically described by stochastic differential equations (SDEs). A fundamental challenge in science and engineering is to determine the governing equations of a complex system from snapshot data. Traditional equation discovery methods often rely on stringent assumptions, such as the availability of the trajectory information or time-series data, and the presumption that the underlying system is deterministic. In this work, we introduce a data-driven, simulation-free framework, called Sparse Identification of Differential Equations from Snapshots (SpIDES), that discovers the governing equations of a complex system from snapshots by utilizing the advanced machine learning techniques to perform three essential steps: probability flow reconstruction, probability density estimation, and Bayesian sparse identification. We validate the effectiveness and robustness of SpIDES by successfully identifying the governing equation of an over-damped Langevin system confined within two potential wells. By extracting interpretable drift and diffusion terms from the SDEs, our framework provides deeper insights into system dynamics, enhances predictive accuracy, and facilitates more effective strategies for managing and simulating stochastic systems.

cs.LG↗

arXiv2024

PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbations

Expert-designed close-ended benchmarks are indispensable in assessing the knowledge capacity of large language models (LLMs). Despite their widespread use, concerns have mounted regarding their reliability due to limited test scenarios and an unavoidable risk of data contamination. To rectify this, we present PertEval, a toolkit devised for in-depth probing of LLMs' knowledge capacity through \textbf{knowledge-invariant perturbations}. These perturbations employ human-like restatement techniques to generate on-the-fly test samples from static benchmarks, meticulously retaining knowledge-critical content while altering irrelevant details. Our toolkit further includes a suite of \textbf{response consistency analyses} that compare performance on raw vs. perturbed test sets to precisely assess LLMs' genuine knowledge capacity. Six representative LLMs are re-evaluated using PertEval. Results reveal significantly inflated performance of the LLMs on raw benchmarks, including an absolute 25.8% overestimation for GPT-4. Additionally, through a nuanced response pattern analysis, we discover that PertEval retains LLMs' uncertainty to specious knowledge, and reveals their potential rote memorization to correct options which leads to overestimated performance. We also find that the detailed response consistency analyses by PertEval could illuminate various weaknesses in existing LLMs' knowledge mastery and guide the development of refinement. Our findings provide insights for advancing more robust and genuinely knowledgeable LLMs. Our code is available at \url{https://github.com/aigc-apps/PertEval}.

cs.CL↗

arXiv2024

CTA-Net: A CNN-Transformer Aggregation Network for Improving Multi-Scale Feature Extraction

Convolutional neural networks (CNNs) and vision transformers (ViTs) have become essential in computer vision for local and global feature extraction. However, aggregating these architectures in existing methods often results in inefficiencies. To address this, the CNN-Transformer Aggregation Network (CTA-Net) was developed. CTA-Net combines CNNs and ViTs, with transformers capturing long-range dependencies and CNNs extracting localized features. This integration enables efficient processing of detailed local and broader contextual information. CTA-Net introduces the Light Weight Multi-Scale Feature Fusion Multi-Head Self-Attention (LMF-MHSA) module for effective multi-scale feature integration with reduced parameters. Additionally, the Reverse Reconstruction CNN-Variants (RRCV) module enhances the embedding of CNNs within the transformer architecture. Extensive experiments on small-scale datasets with fewer than 100,000 samples show that CTA-Net achieves superior performance (TOP-1 Acc 86.76\%), fewer parameters (20.32M), and greater efficiency (FLOPs 2.83B), making it a highly efficient and lightweight solution for visual tasks on small-scale datasets (fewer than 100,000).

cs.CV↗

arXiv2024

Meso-inflationary Peccei-Quinn symmetry breaking with non-minimal coupling

We study a realization of the inflationary scenario where the Peccei-Quinn (PQ) symmetry is spontaneously broken during inflation, facilitated by its non-minimal coupling to gravity. This results in effectively two-field inflation: the early stage is driven by an inflaton field with the PQ symmetry intact, and the later stage is driven by the PQ scalar after its effective mass becomes tachyonic, causing destabilization from the origin. The non-minimal coupling serves the dual purpose of restoring the PQ symmetry during early inflation and flattening the PQ potential post-tachyonic shift, allowing for continued slow roll. We analyze the inflationary background solutions and scalar perturbations, which are amplified at small scales via significant isocurvature perturbations generated near the symmetry-breaking epoch. These perturbations lead to second-order gravitational waves, detectable by next-generation space-based experiments.

hep-th↗

arXiv2024

Exact first passage time distribution for second-order reactions in chemical networks

The first passage time (FPT) is a generic measure that quantifies when a random quantity reaches a specific state. We consider the FTP distribution in nonlinear stochastic biochemical networks, where obtaining exact solutions of the distribution is a challenging problem. Even simple two-particle collisions cause strong nonlinearities that hinder the theoretical determination of the full FPT distribution. Previous research has either focused on analyzing the mean FPT, which provides limited information about a system, or has considered time-consuming stochastic simulations that do not clearly expose causal relationships between parameters and the system's dynamics. This paper presents the first exact theoretical solution of the full FPT distribution in a broad class of chemical reaction networks involving $A + B \rightarrow C$ type of second-order reactions. Our exact theoretical method outperforms stochastic simulations, in terms of computational efficiency, and deviates from approximate analytical solutions. Given the prevalence of bimolecular reactions in biochemical systems, our approach has the potential to enhance the understanding of real-world biochemical processes.

q-bio.MN↗

arXiv2024

Network Structure Governs Drosophila Brain Functionality

How intelligence emerges from living beings has been a fundamental question in neuroscience. However, it remains largely unanswered due to the complex neuronal dynamics and intricate connections between neurons in real neural systems. To address this challenge, we leveraged the largest available adult Drosophila connectome data set, and constructed a comprehensive computational framework based on simplified neuronal activation mechanisms to simulate the observed activation behavior within the connectome. The results revealed that even with rudimentary neuronal activation mechanisms, models grounded in real neural network structures can generate activation patterns strikingly similar to those observed in the actual brain. A significant discovery was the consistency of activation patterns across various neuronal dynamic models. This consistency, achieved with the same network structure, underscores the pivotal role of network topology in neural information processing. These results challenge the prevailing view that solely relies on neuron count or complex individual neuron dynamics. Further analysis demonstrated a near-complete separation of the visual and olfactory systems at the network level. Moreover, we found that the network distance, rather than spatial distance, is the primary determinant of activation patterns. Additionally, our experiments revealed that a reconnect rate of at least 0.1% was sufficient to disrupt the previously observed activation patterns. We also observed synergistic effects between the brain hemispheres: Even with unilateral input stimuli, visual-related neurons in both hemispheres were activated, highlighting the importance of interhemispheric communication. These findings emphasize the crucial role of network structure in neural activation and offer novel insights into the fundamental principles governing brain functionality.

q-bio.NC↗

arXiv2024

First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models

Dynamic activation (DA) techniques, such as DejaVu and MoEfication, have demonstrated their potential to significantly enhance the inference efficiency of large language models (LLMs). However, these techniques often rely on ReLU activation functions or require additional parameters and training to maintain performance. This paper introduces a training-free Threshold-based Dynamic Activation(TDA) method that leverage sequence information to exploit the inherent sparsity of models across various architectures. This method is designed to accelerate generation speed by 18-25\% without significantly compromising task performance, thereby addressing the limitations of existing DA techniques. Moreover, we delve into the root causes of LLM sparsity and theoretically analyze two of its critical features: history-related activation uncertainty and semantic-irrelevant activation inertia. Our comprehensive analyses not only provide a robust theoretical foundation for DA methods but also offer valuable insights to guide future research in optimizing LLMs for greater efficiency and effectiveness.

cs.CL↗

arXiv2024

LARR: Large Language Model Aided Real-time Scene Recommendation with Semantic Understanding

Click-Through Rate (CTR) prediction is crucial for Recommendation System(RS), aiming to provide personalized recommendation services for users in many aspects such as food delivery, e-commerce and so on. However, traditional RS relies on collaborative signals, which lacks semantic understanding to real-time scenes. We also noticed that a major challenge in utilizing Large Language Models (LLMs) for practical recommendation purposes is their efficiency in dealing with long text input. To break through the problems above, we propose Large Language Model Aided Real-time Scene Recommendation(LARR), adopt LLMs for semantic understanding, utilizing real-time scene information in RS without requiring LLM to process the entire real-time scene text directly, thereby enhancing the efficiency of LLM-based CTR modeling. Specifically, recommendation domain-specific knowledge is injected into LLM and then RS employs an aggregation encoder to build real-time scene information from separate LLM's outputs. Firstly, a LLM is continual pretrained on corpus built from recommendation data with the aid of special tokens. Subsequently, the LLM is fine-tuned via contrastive learning on three kinds of sample construction strategies. Through this step, LLM is transformed into a text embedding model. Finally, LLM's separate outputs for different scene features are aggregated by an encoder, aligning to collaborative signals in RS, enhancing the performance of recommendation model.

cs.IR↗

arXiv2024

Aligning Explanations for Recommendation with Rating and Feature via Maximizing Mutual Information

Providing natural language-based explanations to justify recommendations helps to improve users' satisfaction and gain users' trust. However, as current explanation generation methods are commonly trained with an objective to mimic existing user reviews, the generated explanations are often not aligned with the predicted ratings or some important features of the recommended items, and thus, are suboptimal in helping users make informed decision on the recommendation platform. To tackle this problem, we propose a flexible model-agnostic method named MMI (Maximizing Mutual Information) framework to enhance the alignment between the generated natural language explanations and the predicted rating/important item features. Specifically, we propose to use mutual information (MI) as a measure for the alignment and train a neural MI estimator. Then, we treat a well-trained explanation generation model as the backbone model and further fine-tune it through reinforcement learning with guidance from the MI estimator, which rewards a generated explanation that is more aligned with the predicted rating or a pre-defined feature of the recommended item. Experiments on three datasets demonstrate that our MMI framework can boost different backbone models, enabling them to outperform existing baselines in terms of alignment with predicted ratings and item features. Additionally, user studies verify that MI-enhanced explanations indeed facilitate users' decisions and are favorable compared with other baselines due to their better alignment properties.

cs.IR↗

arXiv2024

DEEPEAST technique to enhance power in two-sample tests via the same-attraction function

Data depth has emerged as an invaluable nonparametric measure for the ranking of multivariate samples. The main contribution of depth-based two-sample comparisons is the introduction of the Q statistic (Liu and Singh, 1993), a quality index. Unlike traditional methods, data depth does not require the assumption of normal distributions and adheres to four fundamental properties. Many existing two-sample homogeneity tests, which assess mean and/or scale changes in distributions often suffer from low statistical power or indeterminate asymptotic distributions. To overcome these challenges, we introduced a DEEPEAST (depth-explored same-attraction sample-to-sample central-outward ranking) technique for improving statistical power in two-sample tests via the same-attraction function. We proposed two novel and powerful depth-based test statistics: the sum test statistic and the product test statistic, which are rooted in Q statistics, share a "common attractor" and are applicable across all depth functions. We further proved the asymptotic distribution of these statistics for various depth functions. To assess the performance of power gain, we apply three depth functions: Mahalanobis depth (Liu and Singh, 1993), Spatial depth (Brown, 1958; Gower, 1974), and Projection depth (Liu, 1992). Through two-sample simulations, we have demonstrated that our sum and product statistics exhibit superior power performance, utilizing a strategic block permutation algorithm and compare favourably with popular methods in literature. Our tests are further validated through analysis on Raman spectral data, acquired from cellular and tissue samples, highlighting the effectiveness of the proposed tests highlighting the effective discrimination between health and cancerous samples.

stat.ME↗

arXiv2024

Rubick: Exploiting Job Reconfigurability for Deep Learning Cluster Scheduling

The era of large deep learning models has given rise to advanced training strategies such as 3D parallelism and the ZeRO series. These strategies enable various (re-)configurable execution plans for a training job, which exhibit remarkably different requirements of multiple resource types. Existing cluster scheduling systems, however, treat such reconfigurable training jobs as black boxes: they rely on users to choose execution plans statically, and then make resource allocations without awareness of the chosen plans and their resource requirements. This approach results in mismatches between execution plans and resources, making both training performance and cluster utilization far from optimal. We introduce Rubick, a cluster scheduling system for deep learning training that exploits the reconfigurability to improve job performance and cluster efficiency. Rubick incorporates the job execution planning as a new dimension in cluster scheduling, by continuously reconfiguring jobs' execution plans and tuning multi-resource allocations across jobs jointly. Such a co-optimization is navigated by a performance model that understands the diverse resource requirements and performance characteristics of different jobs and execution plans. Rubick exploits such a model to make performance-aware scheduling decisions to maximize cluster throughput while providing performance guarantees to individual jobs. Evaluations on a 64-GPU high-performance training cluster show that Rubick improves average job completion time and makespan by up to 3.2x and 1.4x, respectively, compared against state-of-the-art systems.

cs.DC↗

arXiv2024

Robust Zero-Shot Crowd Counting and Localization With Adaptive Resolution SAM

The existing crowd counting models require extensive training data, which is time-consuming to annotate. To tackle this issue, we propose a simple yet effective crowd counting method by utilizing the Segment-Everything-Everywhere Model (SEEM), an adaptation of the Segmentation Anything Model (SAM), to generate pseudo-labels for training crowd counting models. However, our initial investigation reveals that SEEM's performance in dense crowd scenes is limited, primarily due to the omission of many persons in high-density areas. To overcome this limitation, we propose an adaptive resolution SEEM to handle the scale variations, occlusions, and overlapping of people within crowd scenes. Alongside this, we introduce a robust localization method, based on Gaussian Mixture Models, for predicting the head positions in the predicted people masks. Given the mask and point pseudo-labels, we propose a robust loss function, which is designed to exclude uncertain regions based on SEEM's predictions, thereby enhancing the training process of the counting networks. Finally, we propose an iterative method for generating pseudo-labels. This method aims at improving the quality of the segmentation masks by identifying more tiny persons in high-density regions, which are often missed in the first pseudo-labeling stage. Overall, our proposed method achieves the best unsupervised performance in crowd counting, while also being comparable results to some supervised methods. This makes it a highly effective and versatile tool for crowd counting, especially in situations where labeled data is not available.

cs.CV↗

arXiv2024

Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs

Prompt ensembling of Large Language Model (LLM) generated category-specific prompts has emerged as an effective method to enhance zero-shot recognition ability of Vision-Language Models (VLMs). To obtain these category-specific prompts, the present methods rely on hand-crafting the prompts to the LLMs for generating VLM prompts for the downstream tasks. However, this requires manually composing these task-specific prompts and still, they might not cover the diverse set of visual concepts and task-specific styles associated with the categories of interest. To effectively take humans out of the loop and completely automate the prompt generation process for zero-shot recognition, we propose Meta-Prompting for Visual Recognition (MPVR). Taking as input only minimal information about the target task, in the form of its short natural language description, and a list of associated class labels, MPVR automatically produces a diverse set of category-specific prompts resulting in a strong zero-shot classifier. MPVR generalizes effectively across various popular zero-shot image recognition benchmarks belonging to widely different domains when tested with multiple LLMs and VLMs. For example, MPVR obtains a zero-shot recognition improvement over CLIP by up to 19.8% and 18.2% (5.0% and 4.5% on average over 20 datasets) leveraging GPT and Mixtral LLMs, respectively

cs.CV↗

arXiv2024

Vision-Language Guidance for LiDAR-based Unsupervised 3D Object Detection

Accurate 3D object detection in LiDAR point clouds is crucial for autonomous driving systems. To achieve state-of-the-art performance, the supervised training of detectors requires large amounts of human-annotated data, which is expensive to obtain and restricted to predefined object categories. To mitigate manual labeling efforts, recent unsupervised object detection approaches generate class-agnostic pseudo-labels for moving objects, subsequently serving as supervision signal to bootstrap a detector. Despite promising results, these approaches do not provide class labels or generalize well to static objects. Furthermore, they are mostly restricted to data containing multiple drives from the same scene or images from a precisely calibrated and synchronized camera setup. To overcome these limitations, we propose a vision-language-guided unsupervised 3D detection approach that operates exclusively on LiDAR point clouds. We transfer CLIP knowledge to classify point clusters of static and moving objects, which we discover by exploiting the inherent spatio-temporal information of LiDAR point clouds for clustering, tracking, as well as box and label refinement. Our approach outperforms state-of-the-art unsupervised 3D object detectors on the Waymo Open Dataset ($+23~\text{AP}_{3D}$) and Argoverse 2 ($+7.9~\text{AP}_{3D}$) and provides class labels not solely based on object size assumptions, marking a significant advancement in the field.

cs.CV↗

arXiv2024

QQQ: Quality Quattuor-Bit Quantization for Large Language Models

Quantization is a proven effective method for compressing large language models. Although popular techniques like W8A8 and W4A16 effectively maintain model performance, they often fail to concurrently speed up the prefill and decoding stages of inference. W4A8 is a promising strategy to accelerate both of them while usually leads to a significant performance degradation. To address these issues, we present QQQ, a Quality Quattuor-bit Quantization method with 4-bit weights and 8-bit activations. QQQ employs adaptive smoothing and Hessian-based compensation, significantly enhancing the performance of quantized models without extensive training. Furthermore, we meticulously engineer W4A8 GEMM kernels to increase inference speed. Our specialized per-channel W4A8 GEMM and per-group W4A8 GEMM achieve impressive speed increases of 3.67$\times$ and 3.29 $\times$ over FP16 GEMM. Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 $\times$, 2.10$\times$, and 1.25$\times$ compared to FP16, W8A8, and W4A16, respectively.

cs.LG↗

arXiv2024

Bistability in spatiotemporal mode-locking with dynamic multimode gain

Three-dimensional (3D) dissipative soliton existed in spatiotemporal mode-locked (STML) multimode fiber laser has been demonstrated to be a promising formalism for generating high-energy femtosecond pulses, which unfortunately exhibit diverse spatiotemporal dynamics that have not been fully understood. Completely modeling the STML multimode fiber lasers can shed new light on the underlying physics of the spatiotemporal dynamics and thus better manipulate the generation of high-quality energic femtosecond pulses, which however is still largely unmet. To this end, here we theoretically investigate a dynamic multimode gain model of the STML multimode fiber laser by exploring the multimode rate equation (MMRE) in the framework of generalized multimode nonlinear Schrödinger equation. Using this dynamic multimode gain model, the attractor dissection theory is revisited to understand the dominant effects that determine the modal composition of 3D dissipative soliton. Specifically, by varying the numerical aperture of the multimode gain fiber (MMGF), different gain dynamics that correspond to distinct types of gain attractors are observed. As a result, two distinguishing STML operation regimes, respectively governed by the multimode gain effect and spatiotemporal saturable absorption, are identified. In the latter regime, especially, 3D dissipative solitons present bistability that there exist bifurcated solutions with two different linearly polarized (LP) mode compositions. To verify the theoretical findings, the experimental implementation shows that the state of STML can be switched between different LP modes, and confirms the presence of bistability. Particularly, the 3D-soliton shaping mechanism that is governed by the multimode gain effect is testified for the first time, to the best of our knowledge.

physics.optics↗

arXiv2024

Adaptive Self-supervised Robust Clustering for Unstructured Data with Unknown Cluster Number

We introduce a novel self-supervised deep clustering approach tailored for unstructured data without requiring prior knowledge of the number of clusters, termed Adaptive Self-supervised Robust Clustering (ASRC). In particular, ASRC adaptively learns the graph structure and edge weights to capture both local and global structural information. The obtained graph enables us to learn clustering-friendly feature representations by an enhanced graph auto-encoder with contrastive learning technique. It further leverages the clustering results adaptively obtained by robust continuous clustering (RCC) to generate prototypes for negative sampling, which can further contribute to promoting consistency among positive pairs and enlarging the gap between positive and negative samples. ASRC obtains the final clustering results by applying RCC to the learned feature representations with their consistent graph structure and edge weights. Extensive experiments conducted on seven benchmark datasets demonstrate the efficacy of ASRC, demonstrating its superior performance over other popular clustering models. Notably, ASRC even outperforms methods that rely on prior knowledge of the number of clusters, highlighting its effectiveness in addressing the challenges of clustering unstructured data.

cs.LG↗

arXiv2024

Adaptive Utilization of Cross-scenario Information for Multi-scenario Recommendation

Recommender system of the e-commerce platform usually serves multiple business scenarios. Multi-scenario Recommendation (MSR) is an important topic that improves ranking performance by leveraging information from different scenarios. Recent methods for MSR mostly construct scenario shared or specific modules to model commonalities and differences among scenarios. However, when the amount of data among scenarios is skewed or data in some scenarios is extremely sparse, it is difficult to learn scenario-specific parameters well. Besides, simple sharing of information from other scenarios may result in a negative transfer. In this paper, we propose a unified model named Cross-Scenario Information Interaction (CSII) to serve all scenarios by a mixture of scenario-dominated experts. Specifically, we propose a novel method to select highly transferable features in data instances. Then, we propose an attention-based aggregator module, which can adaptively extract relative knowledge from cross-scenario. Experiments on the production dataset verify the superiority of our method. Online A/B test in Meituan Waimai APP also shows a significant performance gain, leading to an average improvement in GMV (Gross Merchandise Value) of 1.0% for overall scenarios.

cs.IR↗