arXiv subjects

Junlin Huang

Publications and source records attributed to Junlin Huang.

10 recordsLinked to original sources

arXiv2026

On the Spectral Flattening of Quantized Embeddings

Training Large Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic heavy-tailed spectral nature of linguistic data. By formalizing the connection between Zipfian statistics and random matrix theory, we prove that the power-law decay in the singular value spectra of embeddings is a fundamental requisite for semantic encoding. We derive theoretical bounds showing that uniform quantization introduces a noise floor that disproportionately truncates this spectral tail, which induces spectral flattening and a strictly provable increase in the stable rank of representations. Empirical validation across diverse architectures including GPT-2 and TinyLlama corroborates that this geometric degradation precipitates representational collapse. This work not only quantifies the spectral sensitivity of LLMs but also establishes spectral fidelity as a necessary condition for stable low-bit optimization.

cs.LG↗

arXiv2025

Probing neutrino-night-quark effective scalar interactions from neutrino masses

In this work, we use neutrino masses as a probe of the neutrino-light-quark effective scalar interactions. It is found that neutrinos can acquire masses not only from the usual light quark loop corrections but also from the light quark condensates. The latter contribution has been overlooked in the literature. We show that both contributions are comparable for operators involving $u$ and $d$ quarks, while quark loop corrections dominate for operators involving the $s$ quark. Using the low-energy effective field theory extended with light right-handed neutrinos and matching to chiral perturbation theory, we systematically analyze these contributions, deriving constraints on the corresponding Wilson coefficients from neutrino mass bounds, coherent elastic neutrino-nucleus scattering, and light pseudoscalar meson invisible decays. Our analysis shows that electron neutrino mass measurements provide the most stringent constraints on these scalar couplings, significantly improving upon limits from other observables. The results highlight the importance of including both perturbative and nonperturbative contributions in complete phenomenological analyses of neutrino mass generation mechanisms.

hep-ph↗

arXiv2025

DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization

The growth of large language models (LLMs) increases challenges of accelerating distributed training across multiple GPUs in different data centers. Moreover, concerns about data privacy and data exhaustion have heightened interest in geo-distributed data centers. Communication in geo-distributed data parallel training (DDP) with stochastic gradient descent (S-SGD) is the main bottleneck in low-bandwidth environments. Local SGD mitigates communication overhead by reducing synchronization frequency, and recent studies have successfully applied it to geo-distributedly pre-train LLMs. However, we identify that its model synchronization mechanism prevents overlapping communication and computation, which makes the system lose opportunities to overlap communication and computation. To overcome this limitation, we expand the design space of local SGD by layer-wisely decoupling model synchronization. In each iteration, only some layers are synchronized instead of the entire model after a specific number of iterations. Leveraging this methodology, we introduce DreamDDP, a training framework to accelerate low-bandwidth distributed training with three key innovations: (1) partial local SGD with theoretical assurances of convergence rates comparable to S-SGD; (2) overlapping parameter synchronization with computation without extra GPU memory occupation; (3) identifying and exploiting three properties to schedule the communication and computation to reduce the training time based on fine-grained profiling of layer-wise communication and computation time. Empirical evaluations conducted on 32 GPUs using prominent deep learning models, including ResNet-18, ResNet-50, GPT-2, and Llama-2, demonstrate that DreamDDP enhances the convergence properties of Local SGD (and Adam) and achieves speedups ranging from $1.49\times$ to $3.91\times$ over leading baseline methods.

cs.DC↗

arXiv2024

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning

Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL). However, these methods encounter challenges such as the straggler problem and diminished model performance due to heterogeneous bandwidth and non-IID (Independently and Identically Distributed) data. To address these issues, we introduce a bandwidth-aware compression framework for FL, aimed at improving communication efficiency while mitigating the problems associated with non-IID data. First, our strategy dynamically adjusts compression ratios according to bandwidth, enabling clients to upload their models at a close pace, thus exploiting the otherwise wasted time to transmit more data. Second, we identify the non-overlapped pattern of retained parameters after compression, which results in diminished client update signals due to uniformly averaged weights. Based on this finding, we propose a parameter mask to adjust the client-averaging coefficients at the parameter level, thereby more closely approximating the original updates, and improving the training convergence under heterogeneous environments. Our evaluations reveal that our method significantly boosts model accuracy, with a maximum improvement of 13% over the uncompressed FedAvg. Moreover, it achieves a $3.37\times$ speedup in reaching the target accuracy compared to FedAvg with a Top-K compressor, demonstrating its effectiveness in accelerating convergence with compression. The integration of common compression techniques into our framework further establishes its potential as a versatile foundation for future cross-device, communication-efficient FL research, addressing critical challenges in FL and advancing the field of distributed machine learning.

cs.DC↗

arXiv2024

Spatio-Temporal Field Neural Networks for Air Quality Inference

The air quality inference problem aims to utilize historical data from a limited number of observation sites to infer the air quality index at an unknown location. Considering the sparsity of data due to the high maintenance cost of the stations, good inference algorithms can effectively save the cost and refine the data granularity. While spatio-temporal graph neural networks have made excellent progress on this problem, their non-Euclidean and discrete data structure modeling of reality limits its potential. In this work, we make the first attempt to combine two different spatio-temporal perspectives, fields and graphs, by proposing a new model, Spatio-Temporal Field Neural Network, and its corresponding new framework, Pyramidal Inference. Extensive experiments validate that our model achieves state-of-the-art performance in nationwide air quality inference in the Chinese Mainland, demonstrating the superiority of our proposed model and framework.

cs.LG↗

arXiv2024

Text-only Synthesis for Image Captioning

From paired image-text training to text-only training for image captioning, the pursuit of relaxing the requirements for high-cost and large-scale annotation of good quality data remains consistent. In this paper, we propose Text-only Synthesis for Image Captioning (ToCa), which further advances this relaxation with fewer human labor and less computing time. Specifically, we deconstruct caption text into structures and lexical words, which serve as the fundamental components of the caption. By combining different structures and lexical words as inputs to the large language model, massive captions that contain various patterns of lexical words are generated. This method not only approaches the target domain but also surpasses it by generating new captions, thereby enhancing the zero-shot generalization ability of the model. Considering the different levels of data access in the real world, we define three synthesis scenarios: cross-domain synthesis, in-domain synthesis, and data-efficient synthesis. Experiments in these scenarios demonstrate the generalizability, transferability and practicability of ToCa with a nearly 5 CIDEr improvement for zero-shot cross-domain captioning and a maximum increase of over 20 CIDEr for data-efficient captioning.

cs.CV↗

arXiv2024

Automatic Radio Map Adaptation for Robust Localization with Dynamic Adversarial Learning

Wireless fingerprint-based localization has become one of the most promising technologies for ubiquitous location-aware computing and intelligent location-based services. However, due to RF vulnerability to environmental dynamics over time, continuous radio map updates are time-consuming and infeasible, resulting in severe accuracy degradation. To address this issue, we propose a novel approach of robust localization with dynamic adversarial learning, known as DadLoc which realizes automatic radio map adaptation by incorporating multiple robust factors underlying RF fingerprints to learn the evolving feature representation with the complicated environmental dynamics. DadLoc performs a finer-grained distribution adaptation with the developed dynamic adversarial adaptation network and quantifies the contributions of both global and local distribution adaptation in a dynamics-adaptive manner. Furthermore, we adopt the strategy of prediction uncertainty suppression to conduct source-supervised training, target-unsupervised training, and source-target dynamic adversarial adaptation which can trade off the environment adaptability and the location discriminability of the learned deep representation for safe and effective feature transfer across different environments. With extensive experimental results, the satisfactory accuracy over other comparative schemes demonstrates that the proposed DanLoc can facilitate fingerprint-based localization for wide deployments.

eess.SP↗

arXiv2022

Evaluating Airline Service Quality Through the Comprehensive Text-mining and TOPSIS-VIKOR-AISM Analysis

Service quality rankings are pivotal for maintaining sustainability in the fiercely competitive airline industry. However, prior research in this domain has often fallen short in aspects of sample size, efficiency, and dependability. This study introduces refined insights into this area and establishes a comprehensive, yet highly elucidative, ranking framework. Initially, we employ Latent Semantic Analysis (LSA) to distill principal themes and sentiments from online reviews of 80 airlines. Subsequently, we utilize the SentiWordNet lexicon and the TextBlob package for conducting sentiment analysis based on these reviews. Following this, we construct a hierarchical structure using the computation of compromise solutions, employing an integrated Technique for Order Preference by Similarity to Ideal Solution, vis-\`a-vis Kriterijumska Optimizacija I Kompromisno Resenje-Adversarial Interpretive Structural Model (TOPSIS-VIKOR-AISM) methodology. Beyond aiding consumer decision-making and fostering airline growth, this study contributes novel viewpoints on evaluating the efficacy of airlines and other sectors.

stat.AP↗

arXiv2022

A geometry method for LED mapping

With inputs from RGB-D camera, industrial camera and wheel odometer, in this letter, we propose a geometry-based detecting method, by which the 3-D modulated LED map can be acquired with the aid of visual odometry algorithm from ORB-SLAM2 system when the decoding result of LED-ID is inaccurate. Subsequently, an enhanced cost function is proposed to optimize the mapping result of LEDs. The average 3-D mapping error (8.5cm) is evaluated with a real-world experiment. This work can be viewed as a preliminary work of visible light positioning systems, offering a way to prevent the labor-intensive manual site surveys of LEDs.

eess.IV↗

arXiv2022

Induced superconductivity in magic-angle twisted trilayer graphene through graphene-metal contacts

Magic-angle twisted trilayer graphene (MATTG) recently exhibited robust superconductivity at a higher transition temperature (TC) than the bilayer version. With electric gating from both the top and bottom sides, the superconductivity was found to be closely associated to two conditions: the finite broken mirror symmetry and carrier concentrations between two to three carriers per moiré unite cell. Both conditions may be achieved by graphene-metal contacts where charge transfers and interfacial electric fields are generated to balance work function mismatch. In this study, we explore the superconductivity of MATTG when contacting a metal, through self-consistently solving the interfacial charge transfer with a highly electric-field-dependent band structure of MATTG. The predicted TC of MATTG-metal contacts forms two domes as a function of the work function difference over the interface, with a maximum over 2 K. Our work provides a constructive reference for graphene experiments and industrial applications with graphene-metal and graphene-semiconductor contacts.

cond-mat.mes-hall↗