Structure of connected nested automorphism groups

Abstract

In this article, we describe the maximal unipotent subgroups of $\mathrm{Aut}(X)$, where $X$ is an affine algebraic variety. Every subgroup of this type has a structure analogous to that of the group of triangular automorphisms of $\mathbb{A}^n$. In particular, it is nested, that is, a countable increasing union of algebraic subgroups. We show that a subgroup $G\subset\mathrm{Aut}(X)$ consisting of unipotent elements is closed if and only if it is nested. This implies that a connected nested subgroup of $\mathrm{Aut}(X)$ is closed, thus answering a question posed by Kraft and Zaidenberg (2022). We also extend the recent description of maximal commutative unipotent subgroups of $\mathrm{Aut}(X)$ due to Regeta and van Santen (2024), by providing a direct construction of such subgroups within our approach.