arXiv subjects

Xinyuan Song

Publications and source records attributed to Xinyuan Song.

121 records · Page 7Linked to original sources

arXiv2024

Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI's ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in the high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus: computational, memory, energy, financial, and network resources and their applicability across various stages of an LLM's lifecycle, including architecture design, pretraining, finetuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current sota and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape.

cs.LG↗

arXiv2024

Deep Learning Model Security: Threats and Defenses

Deep learning has transformed AI applications but faces critical security challenges, including adversarial attacks, data poisoning, model theft, and privacy leakage. This survey examines these vulnerabilities, detailing their mechanisms and impact on model integrity and confidentiality. Practical implementations, including adversarial examples, label flipping, and backdoor attacks, are explored alongside defenses such as adversarial training, differential privacy, and federated learning, highlighting their strengths and limitations. Advanced methods like contrastive and self-supervised learning are presented for enhancing robustness. The survey concludes with future directions, emphasizing automated defenses, zero-trust architectures, and the security challenges of large AI models. A balanced approach to performance and security is essential for developing reliable deep learning systems.

cs.CR↗

arXiv2024

From Noise to Nuance: Advances in Deep Generative Image Models

Deep learning-based image generation has undergone a paradigm shift since 2021, marked by fundamental architectural breakthroughs and computational innovations. Through reviewing architectural innovations and empirical results, this paper analyzes the transition from traditional generative methods to advanced architectures, with focus on compute-efficient diffusion models and vision transformer architectures. We examine how recent developments in Stable Diffusion, DALL-E, and consistency models have redefined the capabilities and performance boundaries of image synthesis, while addressing persistent challenges in efficiency and quality. Our analysis focuses on the evolution of latent space representations, cross-attention mechanisms, and parameter-efficient training methodologies that enable accelerated inference under resource constraints. While more efficient training methods enable faster inference, advanced control mechanisms like ControlNet and regional attention systems have simultaneously improved generation precision and content customization. We investigate how enhanced multi-modal understanding and zero-shot generation capabilities are reshaping practical applications across industries. Our analysis demonstrates that despite remarkable advances in generation quality and computational efficiency, critical challenges remain in developing resource-conscious architectures and interpretable generation systems for industrial applications. The paper concludes by mapping promising research directions, including neural architecture optimization and explainable generation frameworks.

cs.CV↗

arXiv2024

A Comprehensive Guide to Explainable AI: From Classical Models to LLMs

Explainable Artificial Intelligence (XAI) addresses the growing need for transparency and interpretability in AI systems, enabling trust and accountability in decision-making processes. This book offers a comprehensive guide to XAI, bridging foundational concepts with advanced methodologies. It explores interpretability in traditional models such as Decision Trees, Linear Regression, and Support Vector Machines, alongside the challenges of explaining deep learning architectures like CNNs, RNNs, and Large Language Models (LLMs), including BERT, GPT, and T5. The book presents practical techniques such as SHAP, LIME, Grad-CAM, counterfactual explanations, and causal inference, supported by Python code examples for real-world applications. Case studies illustrate XAI's role in healthcare, finance, and policymaking, demonstrating its impact on fairness and decision support. The book also covers evaluation metrics for explanation quality, an overview of cutting-edge XAI tools and frameworks, and emerging research directions, such as interpretability in federated learning and ethical AI considerations. Designed for a broad audience, this resource equips readers with the theoretical insights and practical skills needed to master XAI. Hands-on examples and additional resources are available at the companion GitHub repository: https://github.com/Echoslayer/XAI_From_Classical_Models_to_LLMs.

cs.LG↗

arXiv2024

Deep Learning, Machine Learning, Advancing Big Data Analytics and Management

Advancements in artificial intelligence, machine learning, and deep learning have catalyzed the transformation of big data analytics and management into pivotal domains for research and application. This work explores the theoretical foundations, methodological advancements, and practical implementations of these technologies, emphasizing their role in uncovering actionable insights from massive, high-dimensional datasets. The study presents a systematic overview of data preprocessing techniques, including data cleaning, normalization, integration, and dimensionality reduction, to prepare raw data for analysis. Core analytics methodologies such as classification, clustering, regression, and anomaly detection are examined, with a focus on algorithmic innovation and scalability. Furthermore, the text delves into state-of-the-art frameworks for data mining and predictive modeling, highlighting the role of neural networks, support vector machines, and ensemble methods in tackling complex analytical challenges. Special emphasis is placed on the convergence of big data with distributed computing paradigms, including cloud and edge computing, to address challenges in storage, computation, and real-time analytics. The integration of ethical considerations, including data privacy and compliance with global standards, ensures a holistic perspective on data management. Practical applications across healthcare, finance, marketing, and policy-making illustrate the real-world impact of these technologies. Through comprehensive case studies and Python-based implementations, this work equips researchers, practitioners, and data enthusiasts with the tools to navigate the complexities of modern data analytics. It bridges the gap between theory and practice, fostering the development of innovative solutions for managing and leveraging data in the era of artificial intelligence.

cs.LG↗

arXiv2024

Disruptive Atomic Jumps Induce Grain Boundary Stagnation

Grain growth in polycrystalline materials can be impeded by grain boundary (GB) stagnation. Using atomistic simulations, we observed that during GB migration, the disruptive jumps of a few GB atoms can disturb the original ordered collective movement of GB atoms, leading to the stagnation of the entire GB. These disruptive atomic jumps can be activated by both high driving forces and high temperatures, with even jumps of a few atoms capable of causing the stagnation of an entire GB. This mechanism also explains the non-Arrhenius behavior observed in some GBs. Additionally, a large model size could increase the rate of disruptive atomic jumps, and a clear transition in thermal behavior is observed with the increase of the GB size in GBs exhibiting clear thermally activated stagnation. Our further investigation shows that the disruptive atoms involved in these jumps do not differ from other GB atoms in terms of atomic energy, volume, density, local entropy, or Voronoi tessellation, and no "jam transition" was observed in the energy barrier spectra. This fact makes those disruptive jumps challenging to detect. To address this issue, we propose a displacement vector analysis method that effectively identifies these subtle disruptive jumps.

cond-mat.mtrl-sci↗

arXiv2024

Intrinsic Grain Boundary Shear Coupling Tensor

Grain boundary (GB) migration stands as a linchpin process governing microstructural evolution in polycrystalline materials. Over the past decade, the concept of shear coupling, quantified through the shear coupling factor, has transformed our understanding and driven the development of theoretical frameworks for unifying GB behaviors. In this study, we introduced a novel concept of shear coupling strength designed to overcome the limitations of the conventional shear coupling factor, notably its deficiency in conveying "coupling" information. The shear coupling tensor formed by the shear coupling strengths characterizes intrinsic shear coupling properties across diverse GBs and reveals complex dynamics within the GB mobility tensor. The molecular dynamics simulation confirms the symmetry of the GB mobility tensor. This symmetry is inherently built into the shear coupling strength, aligning with an assumption made in previous studies. Additionally, an efficient methodology has been developed for streamlined extraction of both shear coupling and GB mobility tensors from atomistic simulations. This advancement holds the potential to sample GB behavior across extensive datasets, significantly enhancing our ability to predict structure-property relationships within the expansive 5-parameter space of GBs.

cond-mat.mtrl-sci↗

arXiv2024

Unusual acceleration and size effects in grain boundary migration with shear coupling

Grain boundary (GB) migration plays a crucial role in the thermal and mechanical responses of polycrystalline materials, particularly in ultrafine-grained and nano-grained materials exhibiting grain size-dependent properties. This study investigates the migration behaviors of a set of GBs in Ni through atomistic simulations, employing synthetic driving forces and shear stress. Surprisingly, the displacements of some shear-coupling GBs do not follow the widely assumed linear or approximately linear relation with time; instead, they exhibit a noticeable acceleration tendency. Furthermore, as the bicrystal size perpendicular to the GB plane increases, the boundary velocity significantly decreases. These observations are independent of the magnitude and type of driving force but are closely linked to temperature, unique to shear-coupling GBs that display a rise in the kinetic energy component along the shear direction. By adopting a specific boundary condition, the acceleration in migration and size effect can be largely alleviated. However, the continuous rise in kinetic energy persists, leading to the true driving force for GB migration being lower than the applied value. To address this, we propose a technique to extract the true driving force based on a quantitative analysis of the work-energy relation in the bicrystal system. The calculated true mobility reveals that the recently proposed mobility tensor may not be symmetric at relatively large driving forces. These discoveries advance our understanding of GB migration and offer a scheme to extract the true mobility, crucial for meso- and continuum-scale simulations of GB migration-related phenomena such as crack propagation, recrystallization, and grain growth.

cond-mat.mtrl-sci↗

arXiv2023

On the instrumental variable estimation with many weak and invalid instruments

We discuss the fundamental issue of identification in linear instrumental variable (IV) models with unknown IV validity. With the assumption of the "sparsest rule", which is equivalent to the plurality rule but becomes operational in computation algorithms, we investigate and prove the advantages of non-convex penalized approaches over other IV estimators based on two-step selections, in terms of selection consistency and accommodation for individually weak IVs. Furthermore, we propose a surrogate sparsest penalty that aligns with the identification condition and provides oracle sparse structure simultaneously. Desirable theoretical properties are derived for the proposed estimator with weaker IV strength conditions compared to the previous literature. Finite sample properties are demonstrated using simulations and the selection and estimation method is applied to an empirical study concerning the effect of BMI on diastolic blood pressure.

stat.ME↗

arXiv2023

Structured risk model

Income and risk coexist, yet investors are often so focused on chasing high returns that they overlook the potential risks that can lead to high losses. Therefore, risk forecasting and risk control is the cornerstone of investment. To address the challenge, we construct a multi-factor risk model on the basis of the classical multi-factor modeling framework. For the common factors, inspired by Barra Model's factor classification. we adjust the outliers and missing values of factor exposure data, normalize and finally orthogonalize them, before computing factor returns and making further analysis. Factor return covariance matrix and idiosyncratic return variance matrix are essential tools to express stock returns in the multi-factor risk model. Firstly, we calculate the factor return covariance matrix with EWMA. To tackle the time-series autocorrelation of factor returns, we apply Newey-West adjustment. Then we estimate the idiosyncratic return variance matrix in a similar way and make Newey-West adjustment again to solve the time-series autocorrelation problem. Since the return of a single share is sensitive to missing values and outliers, we introduce structural adjustment to improve the matrix.Eventually, we obtain the return covariance matrix among stocks and compute the risk of investment portfolio based on it. Furthermore, we search for optimal portfolio with respect to minimizing risk or maximizing risk-adjusted return with our model. They provide good Sharpe ratio and information ratio for considering both absolute risk and active risk. Hence, the multi-factor risk model is efficient.

stat.AP↗

arXiv2023

Driving force induced transition in thermal behavior of grain boundary migration in Ni

Grain boundary (GB) migration exhibits intriguing anti-thermal behavior (or non-Arrhenius behavior), with the temperature and driving force playing crucial roles. Through atomistic simulations on nickel bicrystals, we investigate the change in GB mobility with variations in both temperature and driving force. Our results reveal that the GB mobility initially increases with temperature and subsequently decreases after reaching the transition temperature (Ttrans), and, notably, Ttrans exhibits a linear relationship with the activation energy (Q) associated with GB migration. By modulating the driving force, we found that the driving force could effectively lower Q, resulting in the shift of Ttrans towards lower temperatures. Additionally, higher driving forces were found to activate more migration modes at lower temperatures, potentially leading to a transition in the thermal behavior of GB migration. Our work supports the existing theoretical models for GB migration based on both classical thermal activation and disconnection nucleation. Furthermore, we refined the existing model by incorporating the influence of the driving force. The modified model can not only describe the effect of driving force on the thermal behavior of GB migration but also accounts for the observed "anti-driving force" phenomenon in GB migration. Our research has the potential to offer valuable insights for investigating realistic GB migration under more intricate constraints and environments.

cond-mat.mtrl-sci↗

arXiv2022

Computing the intrinsic grain boundary mobility tensor

Grain boundary (GB) mobility has been conventionally computed as a single value; however, a recent study has suggested that GB mobility should be expressed as a tensor. In this work, by using atomistic simulations, the concept of GB mobility being applied to the shear direction was re-examined and it is found that it follows the same physical rule as the conventionally defined GB mobility based on the normal direction. The interface random walk method was then used to compute the intrinsic GB mobility tensor at the zero-driving force limit. In order to compute the off-diagonal elements of the intrinsic GB mobility tensor, a shear coupling strength S is introduced in this study, which we believe can better reflect the intrinsic characteristics of a GB for its coupling trend between the normal and shear motion than the widely used shear coupling factor. Furthermore, the effect of temperature and external driving force on the GB mobility tensor, especially on its symmetry, was systematically investigated. It is shown that the GB mobility in either the normal or shear direction can show a non-Arrhenius type dependence on the applied driving force, which is similar to the widely reported non-Arrhenius (or anti-thermal) dependence of GB mobility on temperature. Accordingly, the classical GB migration equation was adapted to describe the diverse variation of GB mobility due to changes in both temperature and driving force.

cond-mat.mtrl-sci↗

arXiv2011

Discounted continuous-time constrained Markov decision processes in Polish spaces

This paper is devoted to studying constrained continuous-time Markov decision processes (MDPs) in the class of randomized policies depending on state histories. The transition rates may be unbounded, the reward and costs are admitted to be unbounded from above and from below, and the state and action spaces are Polish spaces. The optimality criterion to be maximized is the expected discounted rewards, and the constraints can be imposed on the expected discounted costs. First, we give conditions for the nonexplosion of underlying processes and the finiteness of the expected discounted rewards/costs. Second, using a technique of occupation measures, we prove that the constrained optimality of continuous-time MDPs can be transformed to an equivalent (optimality) problem over a class of probability measures. Based on the equivalent problem and a so-called $\bar{w}$-weak convergence of probability measures developed in this paper, we show the existence of a constrained optimal policy. Third, by providing a linear programming formulation of the equivalent problem, we show the solvability of constrained optimal policies. Finally, we use two computable examples to illustrate our main results.

math.PR↗