Searcharxiv⌕ Search

arXiv subjects

James W. Demmel

Publications and source records attributed to James W. Demmel.

11 recordsLinked to original sources

Adaptive Precision Cholesky with Error Guarantees

Modern accelerators increasingly support low-precision formats such as 8-bit floats and integers to speed up machine learning. We show that a Cholesky factorization in which both GEMM and TRSM are implemented using the fixed-point Ozaki 1 scheme satisfies the same backward error bounds as the standard floating-point Cholesky factorization, and that the same holds for a Cholesky factorization computed using a low-precision MicroScaled GEMM. We then use our error analyses to construct an adaptive precision Cholesky factorization that exploits low-precision formats while respecting a prescribed element-wise error bound, as long as the factorization succeeds. We test it in two settings. First, we use it to choose the number of Ozaki 1 slices at each iteration, with SYRK/GEMM and TRSM emulated in INT8 fixed-point arithmetic to attain double-precision accuracy. Second, we use it to build a low-precision Cholesky preconditioner for the conjugate gradient method (PCG), in which SYRK/GEMM take low-precision inputs and accumulate in high precision, with the input format selected adaptively for each panel; if the factorization breaks down, diagonal perturbation still yields a valid preconditioner, at the cost of the error bound guarantees. On an NVIDIA GH200, the adaptive Ozaki 1-based factorization is up to 29% faster than a right-looking Cholesky using a fixed slice count of 7, and up to 69% faster than one using cuBLAS' Automatic Dynamic Precision (ADP), while maintaining the same error guarantees. For PCG, FP8 factorizations with diagonal perturbation form effective preconditioners on the 13 tested SuiteSparse matrices, reducing the number of PCG iterations by up to $12\times$ over CG on the same matrix diagonally scaled by powers of two. For synthetic matrices with arithmetic spectra, and geometric spectra with $κ\le 10^4$, FP8 preconditioners also reduce the number of CG iterations.

math.NA↗

Mixed-Precision Computing for Scientific Discovery: Formats, Co-Design, and Responsible Approximation

Reduced and mixed precision have moved from a niche optimization to a central design axis in scientific computing and engineering, driven by energy constraints, heterogeneous accelerators, and the convergence of simulation and machine learning. This paper organizes the landscape around seven coupled themes---number formats, floating-point emulation, emerging architectures, hardware/software co-design, relation to other approximations, software design, and precision as a multilevel resource ---and, for each theme, synthesizes the state of the art, future directions, and open questions. We emphasize \emph{energy per trusted solution} as the core objective, and we frame \say{recklessly responsible} computing as a pragmatic doctrine: exploit low precision aggressively, but with systematic detection, escalation, and certification pathways.

math.NA↗

Surrogate-based Autotuning for Randomized Sketching Algorithms in Regression Problems

Algorithms from Randomized Numerical Linear Algebra (RandNLA) are known to be effective in handling high-dimensional computational problems, providing high-quality empirical performance as well as strong probabilistic guarantees. However, their practical application is complicated by the fact that the user needs to set various algorithm-specific tuning parameters which are different than those used in traditional NLA. This paper demonstrates how a surrogate-based autotuning approach can be used to address fundamental problems of parameter selection in RandNLA algorithms. In particular, we provide a detailed investigation of surrogate-based autotuning for sketch-and-precondition (SAP) based randomized least squares methods, which have been one of the great success stories in modern RandNLA. Empirical results show that our surrogate-based autotuning approach can achieve near-optimal performance with much less tuning cost than a random search (up to about 4x fewer trials of different parameter configurations). Moreover, while our experiments focus on least squares, our results demonstrate a general-purpose autotuning pipeline applicable to any kind of RandNLA algorithm.

cs.LG↗

Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization

This paper presents a new type of hybrid model for Bayesian optimization (BO) adept at managing mixed variables, encompassing both quantitative (continuous and integer) and qualitative (categorical) types. Our proposed new hybrid models (named hybridM) merge the Monte Carlo Tree Search structure (MCTS) for categorical variables with Gaussian Processes (GP) for continuous ones. hybridM leverages the upper confidence bound tree search (UCTS) for MCTS strategy, showcasing the tree architecture's integration into Bayesian optimization. Our innovations, including dynamic online kernel selection in the surrogate modeling phase and a unique UCTS search strategy, position our hybrid models as an advancement in mixed-variable surrogate models. Numerical experiments underscore the superiority of hybrid models, highlighting their potential in Bayesian optimization.

cs.LG↗

Non-smooth Bayesian Optimization in Tuning Problems

Building surrogate models is one common approach when we attempt to learn unknown black-box functions. Bayesian optimization provides a framework which allows us to build surrogate models based on sequential samples drawn from the function and find the optimum. Tuning algorithmic parameters to optimize the performance of large, complicated "black-box" application codes is a specific important application, which aims at finding the optima of black-box functions. Within the Bayesian optimization framework, the Gaussian process model produces smooth or continuous sample paths. However, the black-box function in the tuning problem is often non-smooth. This difficult tuning problem is worsened by the fact that we usually have limited sequential samples from the black-box function. Motivated by these issues encountered in tuning, we propose a novel additive Gaussian process model called clustered Gaussian process (cGP), where the additive components are induced by clustering. In the examples we studied, the performance can be improved by as much as 90% among repetitive experiments. By using this surrogate model, we want to capture the non-smoothness of the black-box function. In addition to an algorithm for constructing this model, we also apply the model to several artificial and real applications to evaluate it.

cs.LG↗

Multitask and Transfer Learning for Autotuning Exascale Applications

Multitask learning and transfer learning have proven to be useful in the field of machine learning when additional knowledge is available to help a prediction task. We aim at deriving methods following these paradigms for use in autotuning, where the goal is to find the optimal performance parameters of an application treated as a black-box function. We show comparative results with state-of-the-art autotuning techniques. For instance, we observe an average $1.5x$ improvement of the application runtime compared to the OpenTuner and HpBandSter autotuners. We explain how our approaches can be more suitable than some state-of-the-art autotuners for the tuning of any application in general and of expensive exascale applications in particular.

cs.LG↗

LU factorization with panel rank revealing pivoting and its communication avoiding version

We present the LU decomposition with panel rank revealing pivoting (LU_PRRP), an LU factorization algorithm based on strong rank revealing QR panel factorization. LU_PRRP is more stable than Gaussian elimination with partial pivoting (GEPP). Our extensive numerical experiments show that the new factorization scheme is as numerically stable as GEPP in practice, but it is more resistant to pathological cases and easily solves the Wilkinson matrix and the Foster matrix. We also present CALU_PRRP, a communication avoiding version of LU_PRRP that minimizes communication. CALU_PRRP is based on tournament pivoting, with the selection of the pivots at each step of the tournament being performed via strong rank revealing QR factorization. CALU_PRRP is more stable than CALU, the communication avoiding version of GEPP. CALU_PRRP is also more stable in practice and is resistant to pathological cases on which GEPP and CALU fail.

math.NA↗

Minimizing Polynomials Over Semialgebraic Sets

This paper concerns a method for finding the minimum of a polynomial on a semialgebraic set, i.e., a set in $\re^m$ defined by finitely many polynomial equations and inequalities, using the Karush-Kuhn-Tucker (KKT) system and sum of squares (SOS) relaxations. This generalizes results in the recent paper \cite{njw_grad}, which considers minimizing polynomials on algebraic sets, i.e., sets in $\re^m$ defined by finitely many polynomial equations. Most of the theorems and conclusions in \cite{njw_grad} generalize to semialgebraic sets, even in the case where the semialgebraic set is not compact. We discuss the method in some special cases, namely, when the semialgebraic set is contained in the nonnegative orthant $\re^n_+$ or in box constraints $[a,b]_n$. These constraints make the computations more efficient.

math.OC↗