arXiv subjects

Liang Sun

Publications and source records attributed to Liang Sun.

At least 73 records · Page 4Linked to original sources

arXiv2024

Calibration of the Cryogenic Measurement System of a Resonant Haloscope Cavity

Possible light bosonic dark matter interactions with the Standard Model photon have been searched by microwave resonant cavities. In this paper, we demonstrate the cryogenic readout system calibration of a 7.138 GHz copper cavity with a loaded quality factor $Q_l=10^4$, operated at 22 mK temperature based on a dilution refrigerator. Our readout system consists of High Electron Mobility Transistors as cryogenic amplifiers at 4 K, plus room-temperature amplifiers and a spectrum analyzer for signal power detection. We test the system with a superconducting two-level system as a single-photon source in the microwave frequency regime and report an overall 95.6 dB system gain and -71.4 dB attenuation in the cavity's input channel. The effective noise temperature of the measurement system is 7.5 K.

hep-ex↗

arXiv2024

Addressing Concept Shift in Online Time Series Forecasting: Detect-then-Adapt

Online updating of time series forecasting models aims to tackle the challenge of concept drifting by adjusting forecasting models based on streaming data. While numerous algorithms have been developed, most of them focus on model design and updating. In practice, many of these methods struggle with continuous performance regression in the face of accumulated concept drifts over time. To address this limitation, we present a novel approach, Concept \textbf{D}rift \textbf{D}etection an\textbf{D} \textbf{A}daptation (D3A), that first detects drifting conception and then aggressively adapts the current model to the drifted concepts after the detection for rapid adaption. To best harness the utility of historical data for model adaptation, we propose a data augmentation strategy introducing Gaussian noise into existing training instances. It helps mitigate the data distribution gap, a critical factor contributing to train-test performance inconsistency. The significance of our data augmentation process is verified by our theoretical analysis. Our empirical studies across six datasets demonstrate the effectiveness of D3A in improving model adaptation capability. Notably, compared to a simple Temporal Convolutional Network (TCN) baseline, D3A reduces the average Mean Squared Error (MSE) by $43.9\%$. For the state-of-the-art (SOTA) model, the MSE is reduced by $33.3\%$.

cs.LG↗

arXiv2024

Machine learning based nonlocal kinetic energy density functional for simple metals and alloys

Developing an accurate kinetic energy density functional (KEDF) remains a major hurdle in orbital-free density functional theory. We propose a machine learning based physical-constrained nonlocal (MPN) KEDF and implement it with the usage of the bulk-derived local pseudopotentials and plane wave basis sets in the ABACUS package. The MPN KEDF is designed to satisfy three exact physical constraints: the scaling law of electron kinetic energy, the free electron gas limit, and the non-negativity of Pauli energy density. The MPN KEDF is systematically tested for simple metals, including Li, Mg, Al, and 59 alloys. We conclude that incorporating nonlocal information for designing new KEDFs and obeying exact physical constraints are essential to improve the accuracy, transferability, and stability of ML-based KEDF. These results shed new light on the construction of ML-based functionals.

cond-mat.mtrl-sci↗

arXiv2024

Attention as Robust Representation for Time Series Forecasting

Time series forecasting is essential for many practical applications, with the adoption of transformer-based models on the rise due to their impressive performance in NLP and CV. Transformers' key feature, the attention mechanism, dynamically fusing embeddings to enhance data representation, often relegating attention weights to a byproduct role. Yet, time series data, characterized by noise and non-stationarity, poses significant forecasting challenges. Our approach elevates attention weights as the primary representation for time series, capitalizing on the temporal relationships among data points to improve forecasting accuracy. Our study shows that an attention map, structured using global landmarks and local windows, acts as a robust kernel representation for data points, withstanding noise and shifts in distribution. Our method outperforms state-of-the-art models, reducing mean squared error (MSE) in multivariate time series forecasting by a notable 3.6% without altering the core neural network architecture. It serves as a versatile component that can readily replace recent patching based embedding schemes in transformer-based models, boosting their performance.

cs.LG↗

arXiv2024

FusionSF: Fuse Heterogeneous Modalities in a Vector Quantized Framework for Robust Solar Power Forecasting

Accurate solar power forecasting is crucial to integrate photovoltaic plants into the electric grid, schedule and secure the power grid safety. This problem becomes more demanding for those newly installed solar plants which lack sufficient data. Current research predominantly relies on historical solar power data or numerical weather prediction in a single-modality format, ignoring the complementary information provided in different modalities. In this paper, we propose a multi-modality fusion framework to integrate historical power data, numerical weather prediction, and satellite images, significantly improving forecast performance. We introduce a vector quantized framework that aligns modalities with varying information densities, striking a balance between integrating sufficient information and averting model overfitting. Our framework demonstrates strong zero-shot forecasting capability, which is especially useful for those newly installed plants. Moreover, we collect and release a multi-modal solar power (MMSP) dataset from real-world plants to further promote the research of multi-modal solar forecasting algorithms. Our extensive experiments show that our model not only operates with robustness but also boosts accuracy in both zero-shot forecasting and scenarios rich with training data, surpassing leading models. We have incorporated it into our eForecaster platform and deployed it for more than 300 solar plants with a capacity of over 15GW.

cs.LG↗

arXiv2024

Sparse-VQ Transformer: An FFN-Free Framework with Vector Quantization for Enhanced Time Series Forecasting

Time series analysis is vital for numerous applications, and transformers have become increasingly prominent in this domain. Leading methods customize the transformer architecture from NLP and CV, utilizing a patching technique to convert continuous signals into segments. Yet, time series data are uniquely challenging due to significant distribution shifts and intrinsic noise levels. To address these two challenges,we introduce the Sparse Vector Quantized FFN-Free Transformer (Sparse-VQ). Our methodology capitalizes on a sparse vector quantization technique coupled with Reverse Instance Normalization (RevIN) to reduce noise impact and capture sufficient statistics for forecasting, serving as an alternative to the Feed-Forward layer (FFN) in the transformer architecture. Our FFN-free approach trims the parameter count, enhancing computational efficiency and reducing overfitting. Through evaluations across ten benchmark datasets, including the newly introduced CAISO dataset, Sparse-VQ surpasses leading models with a 7.84% and 4.17% decrease in MAE for univariate and multivariate time series forecasting, respectively. Moreover, it can be seamlessly integrated with existing transformer-based models to elevate their performance.

cs.LG↗

arXiv2024

RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies

Time series forecasting is an important and forefront task in many real-world applications. However, most of time series forecasting techniques assume that the training data is clean without anomalies. This assumption is unrealistic since the collected time series data can be contaminated in practice. The forecasting model will be inferior if it is directly trained by time series with anomalies. Thus it is essential to develop methods to automatically learn a robust forecasting model from the contaminated data. In this paper, we first statistically define three types of anomalies, then theoretically and experimentally analyze the loss robustness and sample robustness when these anomalies exist. Based on our analyses, we propose a simple and efficient algorithm to learn a robust forecasting model. Extensive experiments show that our method is highly robust and outperforms all existing approaches. The code is available at https://github.com/haochenglouis/RobustTSF.

cs.LG↗

arXiv2024

Enhancement of cell membrane poration by the antimicrobial peptide Melp5

Melittin, a natural antimicrobial peptide comprising 26 amino acid residues, can kill bacteria by inducing pores in cell membranes. Clinical applications of melittin as an antibiotic require a thorough understanding of its poration mechanism and mutations that enhance its antimicrobial activity. Previous experiments showed Melp5, a variant of melittin with five mutations, exhibits a higher poration ability. However, the mechanism of the enhanced poration ability is not fully understood. Here, we investigated the mechanism by comparing the poration of melittin and Melp5 using coarse-grained (CG) and all-atom (AA) molecular dynamics (MD) simulations. We observe that Melp5 is likely to form a pore with 5 peptides (pentameric), while melittin is likely to form a pore with 4 peptides (tetrameric). Our atomistic MD simulations show that the pentameric pore of Melp5 has a higher water permeability than the tetrameric pore of melittin. We also analyze the stability of the pores of melittin and Melp5 by calculating the interaction energies of the pores. In particular, we investigate the effects of mutant residues on pore stability by calculating electrostatic and LJ interactions. These results should provide insights on the enhanced poration ability of Melp5 and push it toward applications.

physics.bio-ph↗

arXiv2023

DSAF: A Dual-Stage Adaptive Framework for Numerical Weather Prediction Downscaling

While widely recognized as one of the most substantial weather forecasting methodologies, Numerical Weather Prediction (NWP) usually suffers from relatively coarse resolution and inevitable bias due to tempo-spatial discretization, physical parametrization process, and computation limitation. With the roaring growth of deep learning-based techniques, we propose the Dual-Stage Adaptive Framework (DSAF), a novel framework to address regional NWP downscaling and bias correction tasks. DSAF uniquely incorporates adaptive elements in its design to ensure a flexible response to evolving weather conditions. Specifically, NWP downscaling and correction are well-decoupled in the framework and can be applied independently, which strategically guides the optimization trajectory of the model. Utilizing a multi-task learning mechanism and an uncertainty-weighted loss function, DSAF facilitates balanced training across various weather factors. Additionally, our specifically designed attention-centric learnable module effectively integrates geographic information, proficiently managing complex interrelationships. Experimental validation on the ECMWF operational forecast (HRES) and reanalysis (ERA5) archive demonstrates DSAF's superior performance over existing state-of-the-art models and shows substantial improvements when existing models are augmented using our proposed modules. Code is publicly available at https://github.com/pengwei07/DSAF.

physics.ao-ph↗

arXiv2023

Combining stochastic density functional theory with deep potential molecular dynamics to study warm dense matter

In traditional finite-temperature Kohn-Sham density functional theory (KSDFT), the well-known orbitals wall restricts the use of first-principles molecular dynamics methods at extremely high temperatures. However, stochastic density functional theory (SDFT) can overcome the limitation. Recently, SDFT and its related mixed stochastic-deterministic density functional theory, based on the plane-wave basis set, have been implemented in the first-principles electronic structure software ABACUS [Phys. Rev. B 106, 125132(2022)]. In this study, we combine SDFT with the Born-Oppenheimer molecular dynamics (BOMD) method to investigate systems with temperatures ranging from a few tens of eV to 1000 eV. Importantly, we train machine-learning-based interatomic models using the SDFT data and employ these deep potential models to simulate large-scale systems with long trajectories. Consequently, we compute and analyze the structural properties, dynamic properties, and transport coefficients of warm dense matter. The abovementioned methods offer a new approach with first-principles accuracy to tackle various properties of warm dense matter.

physics.plasm-ph↗

arXiv2023

Receding Horizon Optimization with PPUM: An Approach for Autonomous Robot Path Planning in Uncertain Environments

The ability to understand spatial-temporal patterns for crowds of people is crucial for achieving long-term autonomy of mobile robots deployed in human environments. However, traditional historical data-driven memory models are inadequate for handling anomalies, resulting in poor reasoning by robot in estimating the crowd spatial distribution. In this article, a Receding Horizon Optimization (RHO) formulation is proposed that incorporates a Probability-related Partially Updated Memory (PPUM) for robot path planning in crowded environments with uncertainties. The PPUM acts as a memory layer that combines real-time sensor observations with historical knowledge using a weighted evidence fusion theory to improve robot's adaptivity to the dynamic environments. RHO then utilizes the PPUM as a informed knowledge to generate a path that minimizes the likelihood of encountering dense crowds while reducing the cost of local motion planning. The proposed approach provides an innovative solution to the problem of robot's long-term safe interaction with human in uncertain crowded environments. In simulation, the results demonstrate the superior performance of our approach compared to benchmark methods in terms of crowd distribution estimation accuracy, adaptability to anomalies and path planning efficiency.

cs.RO↗

arXiv2023

Interactive Generalized Additive Model and Its Applications in Electric Load Forecasting

Electric load forecasting is an indispensable component of electric power system planning and management. Inaccurate load forecasting may lead to the threat of outages or a waste of energy. Accurate electric load forecasting is challenging when there is limited data or even no data, such as load forecasting in holiday, or under extreme weather conditions. As high-stakes decision-making usually follows after load forecasting, model interpretability is crucial for the adoption of forecasting models. In this paper, we propose an interactive GAM which is not only interpretable but also can incorporate specific domain knowledge in electric power industry for improved performance. This boosting-based GAM leverages piecewise linear functions and can be learned through our efficient algorithm. In both public benchmark and electricity datasets, our interactive GAM outperforms current state-of-the-art methods and demonstrates good generalization ability in the cases of extreme weather events. We launched a user-friendly web-based tool based on interactive GAM and already incorporated it into our eForecaster product, a unified AI platform for electricity forecasting.

cs.LG↗

arXiv2023

One Fits All:Power General Time Series Analysis by Pretrained LM

Although we have witnessed great success of pre-trained models in natural language processing (NLP) and computer vision (CV), limited progress has been made for general time series analysis. Unlike NLP and CV where a unified model can be used to perform different tasks, specially designed approach still dominates in each time series analysis task such as classification, anomaly detection, forecasting, and few-shot learning. The main challenge that blocks the development of pre-trained model for time series analysis is the lack of a large amount of data for training. In this work, we address this challenge by leveraging language or CV models, pre-trained from billions of tokens, for time series analysis. Specifically, we refrain from altering the self-attention and feedforward layers of the residual blocks in the pre-trained language or image model. This model, known as the Frozen Pretrained Transformer (FPT), is evaluated through fine-tuning on all major types of tasks involving time series. Our results demonstrate that pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks, as illustrated in Figure 1. We also found both theoretically and empirically that the self-attention module behaviors similarly to principle component analysis (PCA), an observation that helps explains how transformer bridges the domain gap and a crucial step towards understanding the universality of a pre-trained transformer.The code is publicly available at https://github.com/DAMO-DI-ML/One_Fits_All.

cs.LG↗

arXiv2023

DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection

Time series anomaly detection is critical for a wide range of applications. It aims to identify deviant samples from the normal sample distribution in time series. The most fundamental challenge for this task is to learn a representation map that enables effective discrimination of anomalies. Reconstruction-based methods still dominate, but the representation learning with anomalies might hurt the performance with its large abnormal loss. On the other hand, contrastive learning aims to find a representation that can clearly distinguish any instance from the others, which can bring a more natural and promising representation for time series anomaly detection. In this paper, we propose DCdetector, a multi-scale dual attention contrastive representation learning model. DCdetector utilizes a novel dual attention asymmetric design to create the permutated environment and pure contrastive loss to guide the learning process, thus learning a permutation invariant representation with superior discrimination abilities. Extensive experiments show that DCdetector achieves state-of-the-art results on multiple time series anomaly detection benchmark datasets. Code is publicly available at https://github.com/DAMO-DI-ML/KDD2023-DCdetector.

cs.LG↗

arXiv2023

OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling

Online updating of time series forecasting models aims to address the concept drifting problem by efficiently updating forecasting models based on streaming data. Many algorithms are designed for online time series forecasting, with some exploiting cross-variable dependency while others assume independence among variables. Given every data assumption has its own pros and cons in online time series modeling, we propose \textbf{On}line \textbf{e}nsembling \textbf{Net}work (OneNet). It dynamically updates and combines two models, with one focusing on modeling the dependency across the time dimension and the other on cross-variate dependency. Our method incorporates a reinforcement learning-based approach into the traditional online convex programming framework, allowing for the linear combination of the two models with dynamically adjusted weights. OneNet addresses the main shortcoming of classical online learning methods that tend to be slow in adapting to the concept drift. Empirical results show that OneNet reduces online forecasting error by more than $\mathbf{50\%}$ compared to the State-Of-The-Art (SOTA) method. The code is available at \url{https://github.com/yfzhang114/OneNet}.

cs.LG↗

arXiv2023

GCformer: An Efficient Framework for Accurate and Scalable Long-Term Multivariate Time Series Forecasting

Transformer-based models have emerged as promising tools for time series forecasting. However, these model cannot make accurate prediction for long input time series. On the one hand, they failed to capture global dependencies within time series data. On the other hand, the long input sequence usually leads to large model size and high time complexity. To address these limitations, we present GCformer, which combines a structured global convolutional branch for processing long input sequences with a local Transformer-based branch for capturing short, recent signals. A cohesive framework for a global convolution kernel has been introduced, utilizing three distinct parameterization methods. The selected structured convolutional kernel in the global branch has been specifically crafted with sublinear complexity, thereby allowing for the efficient and effective processing of lengthy and noisy input signals. Empirical studies on six benchmark datasets demonstrate that GCformer outperforms state-of-the-art methods, reducing MSE error in multivariate time series benchmarks by 4.38% and model parameters by 61.92%. In particular, the global convolutional branch can serve as a plug-in block to enhance the performance of other models, with an average improvement of 31.93\%, including various recently published Transformer-based models. Our code is publicly available at https://github.com/zyj-111/GCformer.

cs.LG↗

arXiv2023

Truncated Non-Local Kinetic Energy Density Functionals for Simple Metals and Silicon

Adopting an accurate kinetic energy density functional (KEDF) to characterize the noninteracting kinetic energy within the framework of orbital-free density functional theory (OFDFT) is challenging. We propose a new form of the non-local KEDF with a real-space truncation cutoff that satisfies the uniform electron gas limit and design KEDFs for simple metals and silicon. The new KEDFs are obtained by minimizing a residual function, which contains the differences in the total energy and charge density of several representative systems with respect to the Kohn-Sham DFT results. By systematically testing different cutoffs of the new KEDFs, we find that the cutoff plays a crucial role in determining the properties of metallic Al and semiconductor Si systems. We conclude that the new KEDF with a sufficiently long cutoff performs even better than some representative non-local KEDFs in some aspects, which sheds new light on optimizing the KEDFs in OFDFT to achieve better accuracy.

physics.comp-ph↗

arXiv2023

SaDI: A Self-adaptive Decomposed Interpretable Framework for Electric Load Forecasting under Extreme Events

Accurate prediction of electric load is crucial in power grid planning and management. In this paper, we solve the electric load forecasting problem under extreme events such as scorching heats. One challenge for accurate forecasting is the lack of training samples under extreme conditions. Also load usually changes dramatically in these extreme conditions, which calls for interpretable model to make better decisions. In this paper, we propose a novel forecasting framework, named Self-adaptive Decomposed Interpretable framework~(SaDI), which ensembles long-term trend, short-term trend, and period modelings to capture temporal characteristics in different components. The external variable triggered loss is proposed for the imbalanced learning under extreme events. Furthermore, Generalized Additive Model (GAM) is employed in the framework for desirable interpretability. The experiments on both Central China electric load and public energy meters from buildings show that the proposed SaDI framework achieves average 22.14% improvement compared with the current state-of-the-art algorithms in forecasting under extreme events in terms of daily mean of normalized RMSE. Code, Public datasets, and Appendix are available at: https://doi.org/10.24433/CO.9696980.v1 .

cs.LG↗