arXiv subjects

Ying Zhang

Publications and source records attributed to Ying Zhang.

At least 289 records · Page 16Linked to original sources

arXiv2024

Impact of the Inflation Reduction Act and Carbon Capture on Transportation Electrification for a Net-Zero Western U.S. Grid

The electrification of transportation is critical to mitigate Greenhouse Gas (GHG) emissions. The United States (U.S.) government's Inflation Reduction Act (IRA) of 2022 introduces policies to promote the electrification of transportation. In addition to electrifying transportation, clean energy technologies such as Carbon Capture and Storage (CCS) may play a major role in achieving a net-zero energy system. Utilizing scenarios simulated by the U.S. version of the Global Change Analysis Model (GCAM-USA), we analyze the individual and compound contributions of the IRA and CCS to reach a clean U.S. grid by 2035 and net-zero GHG emissions by 2050. We analyze the contributions based on three metrics: i) transportation electrification rate, ii) transportation fuel mix, and iii) spatio-temporal charging loads. Our findings indicate that the IRA significantly accelerates transportation electrification in the near-term (until 2035). In contrast, CCS technologies, by enabling the continued use of internal combustion vehicles while still advancing torward net-zero, potentially suppresses the rate of transportation electrification in the long-term. This study underscores how policy and technology innovation can interact and sensitivity studies with different combination are essential to characterize the potential contributions of each to the transportation electrification.

eess.SY↗

arXiv2024

First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models

Dynamic activation (DA) techniques, such as DejaVu and MoEfication, have demonstrated their potential to significantly enhance the inference efficiency of large language models (LLMs). However, these techniques often rely on ReLU activation functions or require additional parameters and training to maintain performance. This paper introduces a training-free Threshold-based Dynamic Activation(TDA) method that leverage sequence information to exploit the inherent sparsity of models across various architectures. This method is designed to accelerate generation speed by 18-25\% without significantly compromising task performance, thereby addressing the limitations of existing DA techniques. Moreover, we delve into the root causes of LLM sparsity and theoretically analyze two of its critical features: history-related activation uncertainty and semantic-irrelevant activation inertia. Our comprehensive analyses not only provide a robust theoretical foundation for DA methods but also offer valuable insights to guide future research in optimizing LLMs for greater efficiency and effectiveness.

cs.CL↗

arXiv2024

DePatch: Towards Robust Adversarial Patch for Evading Person Detectors in the Real World

Recent years have seen an increasing interest in physical adversarial attacks, which aim to craft deployable patterns for deceiving deep neural networks, especially for person detectors. However, the adversarial patterns of existing patch-based attacks heavily suffer from the self-coupling issue, where a degradation, caused by physical transformations, in any small patch segment can result in a complete adversarial dysfunction, leading to poor robustness in the complex real world. Upon this observation, we introduce the Decoupled adversarial Patch (DePatch) attack to address the self-coupling issue of adversarial patches. Specifically, we divide the adversarial patch into block-wise segments, and reduce the inter-dependency among these segments through randomly erasing out some segments during the optimization. We further introduce a border shifting operation and a progressive decoupling strategy to improve the overall attack capabilities. Extensive experiments demonstrate the superior performance of our method over other physical adversarial attacks, especially in the real world.

cs.CV↗

arXiv2024

QQQ: Quality Quattuor-Bit Quantization for Large Language Models

Quantization is a proven effective method for compressing large language models. Although popular techniques like W8A8 and W4A16 effectively maintain model performance, they often fail to concurrently speed up the prefill and decoding stages of inference. W4A8 is a promising strategy to accelerate both of them while usually leads to a significant performance degradation. To address these issues, we present QQQ, a Quality Quattuor-bit Quantization method with 4-bit weights and 8-bit activations. QQQ employs adaptive smoothing and Hessian-based compensation, significantly enhancing the performance of quantized models without extensive training. Furthermore, we meticulously engineer W4A8 GEMM kernels to increase inference speed. Our specialized per-channel W4A8 GEMM and per-group W4A8 GEMM achieve impressive speed increases of 3.67$\times$ and 3.29 $\times$ over FP16 GEMM. Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 $\times$, 2.10$\times$, and 1.25$\times$ compared to FP16, W8A8, and W4A16, respectively.

cs.LG↗

arXiv2024

Taylor-Expansion-Based Robust Power Flow in Unbalanced Distribution Systems: A Hybrid Data-Aided Method

Traditional power flow methods often adopt certain assumptions designed for passive balanced distribution systems, thus lacking practicality for unbalanced operation. Moreover, their computation accuracy and efficiency are heavily subject to unknown errors and bad data in measurements or prediction data of distributed energy resources (DERs). To address these issues, this paper proposes a hybrid data-aided robust power flow algorithm in unbalanced distribution systems, which combines Taylor series expansion knowledge with a data-driven regression technique. The proposed method initiates a linearization power flow model to derive an explicitly analytical solution by modified Taylor expansion. To mitigate the approximation loss that surges due to the DER integration and bad data, we further develop a data-aided robust support vector regression approach to estimate the errors efficiently. Comparative analysis in the 13-bus and 123-bus IEEE unbalanced feeders shows that the proposed algorithm achieves superior computational efficiency, with guaranteed accuracy and robustness against outliers.

eess.SY↗

arXiv2024

Paths of A Million People: Extracting Life Trajectories from Wikipedia

The life trajectories of notable people have been studied to pinpoint the times and places of significant events such as birth, death, education, marriage, competition, work, speeches, scientific discoveries, artistic achievements, and battles. Understanding how these individuals interact with others provides valuable insights for broader research into human dynamics. However, the scarcity of trajectory data in terms of volume, density, and inter-person interactions, limits relevant studies from being comprehensive and interactive. We mine millions of biography pages from Wikipedia and tackle the generalization problem stemming from the variety and heterogeneity of the trajectory descriptions. Our ensemble model COSMOS, which combines the idea of semi-supervised learning and contrastive learning, achieves an F1 score of 85.95%. For this task, we also create a hand-curated dataset, WikiLifeTrajectory, consisting of 8,852 (person, time, location) triplets as ground truth. Besides, we perform an empirical analysis on the trajectories of 8,272 historians to demonstrate the validity of the extracted results. To facilitate the research on trajectory extractions and help the analytical studies to construct grand narratives, we make our code, the million-level extracted trajectories, and the WikiLifeTrajectory dataset publicly available.

cs.CL↗

arXiv2024

The first-order phase transition in the neutron star from the deep neural network

This study investigates the first-order phase transition within neutron stars, leveraging the deep neural network (DNN) framework alongside contemporary astronomical measurements. The equation of state (EOS) for neutron stars is delineated in a piecewise polytropic form, with the speed of sound ($c_s$) serving as a pivotal determinant. In the phase transition region, $c_s$ is presumed to be zero, while in other intervals, it is optimized utilizing the DNN. Various onset energy densities of phase transition ($\varepsilon_{pt}$), spanning from $2\varepsilon_0$ to $3\varepsilon_0$ (where $\varepsilon_0$ denotes the energy density at nuclear saturation density), as well as phase transition widths ($Δ\varepsilon$) ranging from $0.5\varepsilon_0$ to $\varepsilon_0$, are examined. Our findings underscore that smaller values of $\varepsilon_{pt}$ lead to a more substantial impact of $Δ\varepsilon$ on neutron star properties, encompassing maximum mass, corresponding radius, tidal deformability, phase transition mass, and trace anomaly. Conversely, when $\varepsilon_{pt}$ exceeds $2.5\varepsilon_0$, the influence of $Δ\varepsilon$ diminishes, resulting in a stiffer EOS compared to scenarios lacking a phase transition. Furthermore, the trace anomaly at high density shifts to negative values upon the commencement of the phase transition. It is noteworthy that the correlations between the average speed of sound at different energy density segments demonstrate a notably weak connection. The discernment of whether a phase transition has occurred with the present observables of neutron stars poses a challenging task.

nucl-th↗

arXiv2024

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present in recent hardware, with FlashAttention-2 achieving only 35% utilization on the H100 GPU. We develop three main techniques to speed up attention on Hopper GPUs: exploiting asynchrony of the Tensor Cores and TMA to (1) overlap overall computation and data movement via warp-specialization and (2) interleave block-wise matmul and softmax operations, and (3) block quantization and incoherent processing that leverages hardware support for FP8 low-precision. We demonstrate that our method, FlashAttention-3, achieves speedup on H100 GPUs by 1.5-2.0$\times$ with FP16 reaching up to 740 TFLOPs/s (75% utilization), and with FP8 reaching close to 1.2 PFLOPs/s. We validate that FP8 FlashAttention-3 achieves 2.6$\times$ lower numerical error than a baseline FP8 attention.

cs.LG↗

arXiv2024

Tactics, Techniques, and Procedures (TTPs) in Interpreted Malware: A Zero-Shot Generation with Large Language Models

Nowadays, the open-source software (OSS) ecosystem suffers from security threats of software supply chain (SSC) attacks. Interpreted OSS malware plays a vital role in SSC attacks, as criminals have an arsenal of attack vectors to deceive users into installing malware and executing malicious activities. In this paper, we introduce tactics, techniques, and procedures (TTPs) proposed by MITRE ATT\&CK into the interpreted malware analysis to characterize different phases of an attack lifecycle. Specifically, we propose GENTTP, a zero-shot approach to extracting a TTP of an interpreted malware package. GENTTP leverages large language models (LLMs) to automatically generate a TTP, where the input is a malicious package, and the output is a deceptive tactic and an execution tactic of attack vectors. To validate the effectiveness of GENTTP, we collect two datasets for evaluation: a dataset with ground truth labels and a large dataset in the wild. Experimental results show that GENTTP can generate TTPs with high accuracy and efficiency. To demonstrate GENTTP's benefits, we build an LLM-based Chatbot from 3,700+ PyPI malware's TTPs. We further conduct a quantitative analysis of malware's TTPs at a large scale. Our main findings include: (1) many OSS malicious packages share a relatively stable TTP, even with the increasing emergence of malware and attack campaigns, (2) a TTP reflects characteristics of a malware-based attack, and (3) an attacker's intent behind the malware is linked to a TTP.

cs.CR↗

arXiv2024

SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning

Parameter-efficient transfer learning (PETL) has emerged as a flourishing research field for adapting large pre-trained models to downstream tasks, greatly reducing trainable parameters while grappling with memory challenges during fine-tuning. To address it, memory-efficient series (METL) avoid backpropagating gradients through the large backbone. However, they compromise by exclusively relying on frozen intermediate outputs and limiting the exhaustive exploration of prior knowledge from pre-trained models. Moreover, the dependency and redundancy between cross-layer features are frequently overlooked, thereby submerging more discriminative representations and causing an inherent performance gap (vs. conventional PETL methods). Hence, we propose an innovative METL strategy called SHERL for resource-limited scenarios to decouple the entire adaptation into two successive and complementary processes. In the early route, intermediate outputs are consolidated via an anti-redundancy operation, enhancing their compatibility for subsequent interactions; thereby in the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead and regulate these fairly flexible features into more adaptive and powerful representations for new domains. Extensive ablations on vision-and-language and language-only tasks show that SHERL combines the strengths of both parameter and memory-efficient techniques, performing on-par or better across diverse architectures with lower memory during fine-tuning. Our code is publicly available at: https://github.com/Paranioar/SHERL.

cs.CV↗

arXiv2024

Efficient Maximal Frequent Group Enumeration in Temporal Bipartite Graphs

Cohesive subgraph mining is a fundamental problem in bipartite graph analysis. In reality, relationships between two types of entities often occur at some specific timestamps, which can be modeled as a temporal bipartite graph. However, the temporal information is widely neglected by previous studies. Moreover, directly extending the existing models may fail to find some critical groups in temporal bipartite graphs, which appear in a unilateral (i.e., one-layer) form. To fill the gap, in this paper, we propose a novel model, called maximal λ-frequency group (MFG). Given a temporal bipartite graph G=(U,V,E), a vertex set V_S \subseteq V is an MFG if i) there are no less than λtimestamps, at each of which V_S can form a (t_U,t_V)-biclique with some vertices in U at the corresponding snapshot, and ii) it is maximal. To solve the problem, a filter-and-verification (FilterV) method is proposed based on the Bron-Kerbosch framework, incorporating novel filtering techniques to reduce the search space and array-based strategy to accelerate the frequency and maximality verification. Nevertheless, the cost of frequency verification in each valid candidate set computation and maximality check could limit the scalability of FilterV to larger graphs. Therefore, we further develop a novel verification-free (VFree) approach by leveraging the advanced dynamic counting structure proposed. Theoretically, we prove that VFree can reduce the cost of each valid candidate set computation in FilterV by a factor of O(|V|). Furthermore, VFree can avoid the explicit maximality verification because of the developed search paradigm. Finally, comprehensive experiments on 15 real-world graphs are conducted to demonstrate the efficiency and effectiveness of the proposed techniques and model.

cs.DB↗

arXiv2024

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical model BERT, they could be further improved in two aspects: the relation-level knowledge could be further explored to improve model performance; and the setting of student attention head number could be more flexible to decrease inference time. Therefore, we are motivated to propose a novel knowledge distillation method MLKD-BERT to distill multi-level knowledge in teacher-student framework. Extensive experiments on GLUE benchmark and extractive question answering tasks demonstrate that our method outperforms state-of-the-art knowledge distillation methods on BERT. In addition, MLKD-BERT can flexibly set student attention head number, allowing for substantial inference time decrease with little performance drop.

cs.CL↗

arXiv2024

Contrast then Memorize: Semantic Neighbor Retrieval-Enhanced Inductive Multimodal Knowledge Graph Completion

A large number of studies have emerged for Multimodal Knowledge Graph Completion (MKGC) to predict the missing links in MKGs. However, fewer studies have been proposed to study the inductive MKGC (IMKGC) involving emerging entities unseen during training. Existing inductive approaches focus on learning textual entity representations, which neglect rich semantic information in visual modality. Moreover, they focus on aggregating structural neighbors from existing KGs, which of emerging entities are usually limited. However, the semantic neighbors are decoupled from the topology linkage and usually imply the true target entity. In this paper, we propose the IMKGC task and a semantic neighbor retrieval-enhanced IMKGC framework CMR, where the contrast brings the helpful semantic neighbors close, and then the memorize supports semantic neighbor retrieval to enhance inference. Specifically, we first propose a unified cross-modal contrastive learning to simultaneously capture the textual-visual and textual-textual correlations of query-entity pairs in a unified representation space. The contrastive learning increases the similarity of positive query-entity pairs, therefore making the representations of helpful semantic neighbors close. Then, we explicitly memorize the knowledge representations to support the semantic neighbor retrieval. At test time, we retrieve the nearest semantic neighbors and interpolate them to the query-entity similarity distribution to augment the final prediction. Extensive experiments validate the effectiveness of CMR on three inductive MKGC datasets. Codes are available at https://github.com/OreOZhao/CMR.

cs.MM↗

arXiv2024

Langevin dynamics based algorithm e-TH$\varepsilon$O POULA for stochastic optimization problems with discontinuous stochastic gradient

We introduce a new Langevin dynamics based algorithm, called e-TH$\varepsilon$O POULA, to solve optimization problems with discontinuous stochastic gradients which naturally appear in real-world applications such as quantile estimation, vector quantization, CVaR minimization, and regularized optimization problems involving ReLU neural networks. We demonstrate both theoretically and numerically the applicability of the e-TH$\varepsilon$O POULA algorithm. More precisely, under the conditions that the stochastic gradient is locally Lipschitz in average and satisfies a certain convexity at infinity condition, we establish non-asymptotic error bounds for e-TH$\varepsilon$O POULA in Wasserstein distances and provide a non-asymptotic estimate for the expected excess risk, which can be controlled to be arbitrarily small. Three key applications in finance and insurance are provided, namely, multi-period portfolio optimization, transfer learning in multi-period portfolio optimization, and insurance claim prediction, which involve neural networks with (Leaky)-ReLU activation functions. Numerical experiments conducted using real-world datasets illustrate the superior empirical performance of e-TH$\varepsilon$O POULA compared to SGLD, TUSLA, ADAM, and AMSGrad in terms of model accuracy.

math.OC↗

arXiv2024

High-density single-atom electrocatalytic centers on two-dimensional topological platinum tellurides with Te-vacancy superstructure

Chemical activation of the intrinsically inert basal planes of transition metal dichalcogenides (TMDs) is crucial for developing high-efficiency electrocatalysts for energy technology applications. Here we report the discovery of an efficient TMD-based topological catalyst for hydrogen evolution reaction (HER), containing high-density single-atom reactive centers on a few-layer (7x7)-PtTe2-x superstructure with a Te-vacancy density of x. Compared with pristine Pt(111), PtTe2, and (2x2)-PtTe2-x, (7x7)-PtTe2-x exhibits superior HER performance owing to its substantially increased density of undercoordinated Pt sites, and displays exceptional catalytic stability when operating at high current densities. Our first-principles calculations confirm that multiple types of undercoordinated Pt sites on (7x7)-PtTe2-x exhibit favorable hydrogen adsorption Gibbs free energies, and that the reactive sites can further increase their population upon increasing hydrogen coverage. Both the (2x2)- and (7x7)-PtTe2-x are also shown to possess nontrivial band topology with robust edge states that may further facilitate HER.

cond-mat.mtrl-sci↗

arXiv2024

CryptoGPT: a 7B model rivaling GPT-4 in the task of analyzing and classifying real-time financial news

CryptoGPT: a 7B model competing with GPT-4 in a specific task -- The Impact of Automatic Annotation and Strategic Fine-Tuning via QLoRAIn this article, we present a method aimed at refining a dedicated LLM of reasonable quality with limited resources in an industrial setting via CryptoGPT. It is an LLM designed for financial news analysis for the cryptocurrency market in real-time. This project was launched in an industrial context. This model allows not only for the classification of financial information but also for providing comprehensive analysis. We refined different LLMs of the same size such as Mistral-7B and LLama-7B using semi-automatic annotation and compared them with various LLMs such as GPT-3.5 and GPT-4. Our goal is to find a balance among several needs: 1. Protecting data (by avoiding their transfer to external servers), 2. Limiting annotation cost and time, 3. Controlling the model's size (to manage deployment costs), and 4. Maintaining better analysis quality.

cs.AI↗

arXiv2024

AutoFirm: Automatically Identifying Reused Libraries inside IoT Firmware at Large-Scale

The Internet of Things (IoT) has become indispensable to our daily lives and work. Unfortunately, developers often reuse software libraries in the IoT firmware, leading to a major security concern. If vulnerabilities or insecure versions of these libraries go unpatched, a massive number of IoT devices can be impacted. In this paper, we propose the AutoFirm, an automated tool for detecting reused libraries in IoT firmware at a large scale. Specifically, AutoFirm leverages the syntax information (library name and version) to determine whether IoT firmware reuses the libraries. We conduct a large-scale empirical study of reused libraries of IoT firmware, investigating more than 6,900+ firmware and 2,700+ distinct vulnerabilities affecting 11,300+ vulnerable versions from 349 open-source software libraries. Leveraging this diverse information set, we conduct a qualitative assessment of vulnerable library versions to understand security gaps and the misplaced trust of libraries in IoT firmware. Our research reveals that: manufacturers neglected to update outdated libraries for IoT firmware in 67.3\% of cases; on average, outdated libraries persisted for over 1.34 years prior to remediation; vulnerabilities of software libraries have posed server threats to widespread IoT devices.

cs.CR↗

arXiv2024

Non-asymptotic convergence bounds for modified tamed unadjusted Langevin algorithm in non-convex setting

We consider the problem of sampling from a high-dimensional target distribution $π_β$ on $\mathbb{R}^d$ with density proportional to $θ\mapsto e^{-βU(θ)}$ using explicit numerical schemes based on discretising the Langevin stochastic differential equation (SDE). In recent literature, taming has been proposed and studied as a method for ensuring stability of Langevin-based numerical schemes in the case of super-linearly growing drift coefficients for the Langevin SDE. In particular, the Tamed Unadjusted Langevin Algorithm (TULA) was proposed in [Bro+19] to sample from such target distributions with the gradient of the potential $U$ being super-linearly growing. However, theoretical guarantees in Wasserstein distances for Langevin-based algorithms have traditionally been derived assuming strong convexity of the potential $U$. In this paper, we propose a novel taming factor and derive, under a setting with possibly non-convex potential $U$ and super-linearly growing gradient of $U$, non-asymptotic theoretical bounds in Wasserstein-1 and Wasserstein-2 distances between the law of our algorithm, which we name the modified Tamed Unadjusted Langevin Algorithm (mTULA), and the target distribution $π_β$. We obtain respective rates of convergence $\mathcal{O}(λ)$ and $\mathcal{O}(λ^{1/2})$ in Wasserstein-1 and Wasserstein-2 distances for the discretisation error of mTULA in step size $λ$. High-dimensional numerical simulations which support our theoretical findings are presented to showcase the applicability of our algorithm.

math.PR↗