arXiv subjects

Mahmoud Safari

Publications and source records attributed to Mahmoud Safari.

At least 19 recordsLinked to original sources

arXiv2026

SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression

Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their deployment is constrained by substantial memory and compute requirements. Low-rank compression via singular value decomposition (SVD) is an effective remedy, but existing methods focus on how to factorize and which components to keep. We introduce SVD-Surgeon, a training-free method that brings the Optimal Brain Surgeon (OBS) framework to the singular-value basis. Treating each singular value as a parameter, it computes a closed-form update of the retained singular values that compensates, to second order in the model loss, for those removed by truncation. The same analysis yields a saliency for choosing which values to prune. As it operates directly on the singular-value factorization, SVD-Surgeon can be layered on top of existing SVD compressors. Applied to SVD-LLM, a leading SVD-based method, it improves the perplexity-compression trade-off on the OPT family and LLaMA 2-7B without any retraining.

cs.LG↗

arXiv2025

Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics

Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by variants such as LeakyReLU, PReLU, and ELU that better handle negative neuron outputs. Recently, self-gated activations like GELU and Swish have emerged as state-of-the-art alternatives, leveraging their smoothness to ensure stable gradient flow and prevent neuron inactivity. In this work, we introduce the Gompertz Linear Unit (GoLU), a novel self-gated activation function defined as $\mathrm{GoLU}(x) = x \, \mathrm{Gompertz}(x)$, where $\mathrm{Gompertz}(x) = e^{-e^{-x}}$. The GoLU activation leverages the right-skewed asymmetry in the Gompertz function to reduce variance in the latent space more effectively compared to GELU and Swish, while preserving robust gradient flow. Extensive experiments across diverse tasks, including Image Classification, Language Modeling, Semantic Segmentation, Object Detection, Instance Segmentation, and Diffusion, highlight GoLU's superior performance relative to state-of-the-art activation functions, establishing GoLU as a robust alternative to existing activation functions.

cs.LG↗

arXiv2024

Efficient Search for Customized Activation Functions with Gradient Descent

Different activation functions work best for different deep learning models. To exploit this, we leverage recent advancements in gradient-based search techniques for neural architectures to efficiently identify high-performing activation functions for a given application. We propose a fine-grained search cell that combines basic mathematical operations to model activation functions, allowing for the exploration of novel activations. Our approach enables the identification of specialized activations, leading to improved performance in every model we tried, from image classification to language models. Moreover, the identified activations exhibit strong transferability to larger models of the same type, as well as new datasets. Importantly, our automated process for creating customized activation functions is orders of magnitude more efficient than previous approaches. It can easily be applied on top of arbitrary deep learning pipelines and thus offers a promising practical avenue for enhancing deep learning architectures.

cs.LG↗

arXiv2024

Surprisingly Strong Performance Prediction with Neural Graph Features

Performance prediction has been a key part of the neural architecture search (NAS) process, allowing to speed up NAS algorithms by avoiding resource-consuming network training. Although many performance predictors correlate well with ground truth performance, they require training data in the form of trained networks. Recently, zero-cost proxies have been proposed as an efficient method to estimate network performance without any training. However, they are still poorly understood, exhibit biases with network properties, and their performance is limited. Inspired by the drawbacks of zero-cost proxies, we propose neural graph features (GRAF), simple to compute properties of architectural graphs. GRAF offers fast and interpretable performance prediction while outperforming zero-cost proxies and other common encodings. In combination with other zero-cost proxies, GRAF outperforms most existing performance predictors at a fraction of the cost.

cs.LG↗

arXiv2023

Weight-Entanglement Meets Gradient-Based Neural Architecture Search

Weight sharing is a fundamental concept in neural architecture search (NAS), enabling gradient-based methods to explore cell-based architectural spaces significantly faster than traditional black-box approaches. In parallel, weight-entanglement has emerged as a technique for more intricate parameter sharing amongst macro-architectural spaces. Since weight-entanglement is not directly compatible with gradient-based NAS methods, these two paradigms have largely developed independently in parallel sub-communities. This paper aims to bridge the gap between these sub-communities by proposing a novel scheme to adapt gradient-based methods for weight-entangled spaces. This enables us to conduct an in-depth comparative assessment and analysis of the performance of gradient-based NAS in weight-entangled search spaces. Our findings reveal that this integration of weight-entanglement and gradient-based NAS brings forth the various benefits of gradient-based methods, while preserving the memory efficiency of weight-entangled spaces. The code for our work is openly accessible https://github.com/automl/TangleNAS.

cs.LG↗

arXiv2023

Neural Architecture Search: Insights from 1000 Papers

In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, and reinforcement learning. Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas. Neural architecture search (NAS), the process of automating the design of neural architectures for a given task, is an inevitable next step in automating machine learning and has already outpaced the best human-designed architectures on many tasks. In the past few years, research in NAS has been progressing rapidly, with over 1000 papers released since 2020 (Deng and Lindauer, 2021). In this survey, we provide an organized and comprehensive guide to neural architecture search. We give a taxonomy of search spaces, algorithms, and speedup techniques, and we discuss resources such as benchmarks, best practices, other surveys, and open-source libraries.

cs.LG↗

arXiv2022

NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies

Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS). Recent work has shown that these techniques show great promise, but certain aspects, such as evaluating and exploiting their complementary strengths, are under-studied. In this work, we create NAS-Bench-Suite: we evaluate 13 ZC proxies across 28 tasks, creating by far the largest dataset (and unified codebase) for ZC proxies, enabling orders-of-magnitude faster experiments on ZC proxies, while avoiding confounding factors stemming from different implementations. To demonstrate the usefulness of NAS-Bench-Suite, we run a large-scale analysis of ZC proxies, including a bias analysis, and the first information-theoretic analysis which concludes that ZC proxies capture substantial complementary information. Motivated by these findings, we present a procedure to improve the performance of ZC proxies by reducing biases such as cell size, and we also show that incorporating all 13 ZC proxies into the surrogate models used by NAS algorithms can improve their predictive performance by up to 42%. Our code and datasets are available at https://github.com/automl/naslib/tree/zerocost.

cs.LG↗

arXiv2022

NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy

The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS). Although they have been widely adopted and used to tune real-world NAS algorithms, these benchmarks are limited to small search spaces and focus solely on image classification. Recently, several new NAS benchmarks have been introduced that cover significantly larger search spaces over a wide range of tasks, including object detection, speech recognition, and natural language processing. However, substantial differences among these NAS benchmarks have so far prevented their widespread adoption, limiting researchers to using just a few benchmarks. In this work, we present an in-depth analysis of popular NAS algorithms and performance prediction methods across 25 different combinations of search spaces and datasets, finding that many conclusions drawn from a few NAS benchmarks do not generalize to other benchmarks. To help remedy this problem, we introduce NAS-Bench-Suite, a comprehensive and extensible collection of NAS benchmarks, accessible through a unified interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research. Our code is available at https://github.com/automl/naslib.

cs.LG↗

arXiv2022

Scale and Conformal Invariance in Higher Derivative Shift Symmetric Theories

The critical behavior of infinite families of shift symmetric interacting theories with higher derivative kinetic terms (non unitary) is considered. Single scalar theories with shift symmetry are classified according to their upper critical dimensions and studied at the leading non trivial order in perturbation theory. For two infinite families, one with quartic and one with cubic interactions, beta functions, criticality conditions and universal anomalous dimensions are computed. At the order considered, the cubic theories enjoy a one loop non renormalization of the vertex, so that the beta function depends non trivially only on the anomalous dimension. The trace of the energy momentum tensor is also investigated and it is shown that these two families of QFTs are conformally invariant at the fixed point of the RG flow.

hep-th↗

arXiv2020

SuperCoder: Program Learning Under Noisy Conditions From Superposition of States

We propose a new method of program learning in a Domain Specific Language (DSL) which is based on gradient descent with no direct search. The first component of our method is a probabilistic representation of the DSL variables. At each timestep in the program sequence, different DSL functions are applied on the DSL variables with a certain probability, leading to different possible outcomes. Rather than handling all these outputs separately, whose number grows exponentially with each timestep, we collect them into a superposition of variables which captures the information in a single, but fuzzy, state. This state is to be contrasted at the final timestep with the ground-truth output, through a loss function. The second component of our method is an attention-based recurrent neural network, which provides an appropriate initialization point for the gradient descent that optimizes the probabilistic representation. The method we have developed surpasses the state-of-the-art for synthesising long programs and is able to learn programs under noise.

cs.LG↗

arXiv2020

A multicritical Landau-Potts field theory

We investigate a perturbatively renormalizable $S_{q}$ invariant model with $N=q-1$ scalar field components below the upper critical dimension $d_c=\frac{10}{3}$. Our results hint at the existence of multicritical generalizations of the critical models of spanning random clusters and percolations in three dimensions. We also discuss the role of our multicritical model in a conjecture that involves the separation of first and second order phases in the $(d,q)$ diagram of the Potts model.

cond-mat.stat-mech↗

arXiv2020

Crossover exponents, fractal dimensions and logarithms in Landau-Potts field theories

We compute the crossover exponents of all quadratic and cubic deformations of critical field theories with permutation symmetry $S_q$ in $d=6-ε$ (Landau-Potts field theories) and $d=4-ε$ (hypertetrahedral models) up to three loops.We use our results to determine the $ε$-expansion of the fractal dimension of critical clusters in the most interesting cases, which include spanning trees and forests ($q\to0$), and bond percolations ($q\to1$). We also explicitly verify several expected degeneracies in the spectrum of relevant operators for natural values of $q$ upon analytic continuation, which are linked to logarithmic corrections of CFT correlators, and use the $ε$-expansion to determine the universal coefficients of such logarithms.

cond-mat.stat-mech↗

arXiv2020

On critical models with $N\leq 4$ scalars in $d=4-ε$

We adopt a combination of analytical and numerical methods to study the renormalization group flow of the most general field theory with quartic interaction in $d=4-ε$ with $N=3$ and $N=4$ scalars. For $N=3$, we find that it admits only three nondecomposable critical points: the Wilson-Fisher with $O(3)$ symmetry, the cubic with $H_3=(\mathbb{Z}_2)^3\rtimes S_3$ symmetry, and the biconical with $O(2)\times \mathbb{Z}_2$. For $N=4$, our analysis reveals the existence of new nontrivial solutions with discrete symmetries and with up to three distinct field anomalous dimensions.

hep-th↗

arXiv2020

Symmetry and universality of multi-field interactions in $6-ε$ dimensions

We outline a general strategy developed for the analysis of critical models, which we apply to obtain a heuristic classification of all universality classes with up to three field-theoretical scalar order parameters in $d=6-ε$ dimensions. As expected by the paradigm of universality, each class is uniquely characterized by its symmetry group and by a set of its scaling properties, neither of which are built-in by the formalism but instead emerge nontrivially as outputs of our computations. For three fields, we find several solutions mostly with discrete symmetries. These are nontrivial conformal field theory candidates in less than six dimensions, one of which is a new perturbatively unitary critical model.

hep-th↗

arXiv2019

Leading order CFT analysis of multi-scalar theories in d>2

We investigate multi-field multicritical scalar theories using CFT constraints on two- and three-point functions combined with the Schwinger-Dyson equation. This is done in general and without assuming any symmetry for the models, which we just define to admit a Landau-Ginzburg description that includes the most general critical interactions built from monomials of the form $ϕ_{i_1} \cdots ϕ_{i_m}$. For all such models we analyze to the leading order of the $ε$-expansion the anomalous dimensions of the fields and those of the composite quadratic operators. For models with even $m$ we extend the analysis to an infinite tower of composite operators of arbitrary order. The results are supplemented by the computation of some families of structure constants. We also find the equations which constrain the nontrivial critical theories at leading order and show that they coincide with the ones obtained with functional perturbative RG methods. This is done for the case $m=3$ as well as for all the even models. We ultimately specialize to $S_q$ symmetric models, which are related to the $q$-state Potts universality class, and focus on three realizations appearing below the upper critical dimensions $6$, $4$ and $\frac{10}{3}$, which can thus be nontrivial CFTs in three dimensions.

hep-th↗

arXiv2018

Uncovering novel phase structures in $\Box^k$ scalar theories with the renormalization group

We present a detailed version of our recent work on the renormalization group approach to multicritical scalar theories with higher derivative kinetic term of the form $ϕ(-\Box)^kϕ$ and upper critical dimension $d_c = 2nk/(n-1)$. Depending on whether the numbers $k$ and $n$ have a common divisor two classes of theories have been distinguished which show qualitatively different features. For coprime $k$ and $n-1$ the theory admits a Wilson-Fisher type fixed point with a marginal interaction $ϕ^{2n}$. We derive in this case the renormalization group equations of the potential at the functional level and compute the scaling dimensions and some OPE coefficients, mostly at leading order in $ε$. While giving new results, the critical data we provide are compared, when possible, and accord with a recent alternative approach using the analytic structure of conformal blocks. Instead when $k$ and $n-1$ have a common divisor we unveil a novel interacting structure at criticality. In this case the phase diagram is more involved as other operators come into play at the scale invariant point. $\Box^2$ theories with odd $n$, which fall in this class, are analyzed in detail. Using the RG flows that are derived at quadratic level in the couplings it is shown that a derivative interaction is unavoidable at the critical point. In particular there is an infrared fixed point with a pure derivative interaction at which we compute the scaling dimensions. For the particular example of $\Box^2$ theory in $d_c=6$ we include some cubic corrections to the flow of the potential which enable us to compute some OPE coefficients as well.

hep-th↗

arXiv2018

Multi-critical $\square^k$ scalar theories: A perturbative RG approach with $ε$-expansion

We employ perturbative RG and $ε$-expansion to study multi-critical single-scalar field theories with higher derivative kinetic terms of the form $ϕ(-\Box)^kϕ$. We focus on those with a $\mathbb{Z}_2$-symmetric critical point which are characterized by an upper critical dimension $d_c=2 n k/(n-1)$ accumulating at even integers. We distinguish two types of theories depending on whether or not the numbers $k$ and $n-1$ are relatively prime. When they are, the theory admits a local potential approximation. In this case we present the beta functional of the potential and use this to calculate some anomalous dimensions and OPE coefficients. These confirm some CFT data obtained using conformal block techniques, while giving new results. In the second case where $k$ and $n-1$ have a common divisor, the theories show a much richer structure induced by the presence of derivative operators. We study the case $k=2$ with odd values of $n$, which fall in the second class, and calculate the functional flows and spectrum. These theories have a phase diagram characterized at leading order in $ε$ by four fixed points which apart from the Gaussian UV fixed point include an IR fixed point with purely derivative interactions.

hep-th↗

arXiv2018

Functional perturbative RG and CFT data in the $ε$-expansion

We show how the use of standard perturbative RG in dimensional regularization allows for a renormalization group based computation of both the spectrum and a family of coefficients of the operator product expansion (OPE) for a given universality class. The task is greatly simplified by a straightforward generalization of perturbation theory to a functional perturbative RG approach. We illustrate our procedure in the $ε$-expansion by obtaining the next-to-leading corrections for the spectrum and the leading corrections for the OPE coefficients of Ising and Lee-Yang universality classes and then give several results for the whole family of renormalizable multicritical models $ϕ^{2n}$. Whenever comparison is possible our RG results explicitly match the ones recently derived in CFT frameworks.

hep-th↗