Annihilator varieties of distinguished modules of reductive Lie algebras

Abstract

We provide a micro-local necessary condition for distinction of admissible representations of real reductive groups in the context of spherical pairs. Let $\bf G$ be a complex algebraic reductive group, and $\bf H\subset G$ be a spherical algebraic subgroup. Let $\mathfrak{g},\mathfrak{h}$ denote the Lie algebras of $\bf G$ and $\bf H$, and let $\mathfrak{h}^{\bot}$ denote the annihilator of $\mathfrak{h}$ in $\mathfrak{g}^*$. A $\mathfrak{g}$-module is called $\mathfrak{h}$-distinguished if it admits a non-zero $\mathfrak{h}$-invariant functional. We show that the maximal $\bf G$-orbit in the annihilator variety of any irreducible $\mathfrak{h}$-distinguished $\mathfrak{g}$-module intersects $\mathfrak{h}^{\bot}$. This generalizes a result of Vogan. We apply this to Casselman-Wallach representations of real reductive groups to obtain information on branching problems, translation functors and Jacquet modules. Further, we prove in many cases that as suggested by Prasad, if $H$ is a symmetric subgroup of a real reductive group $G$, the existence of a tempered $H$-distinguished representation of $G$ implies the existence of a generic $H$-distinguished representation of $G$. Many models studied in the theory of automorphic forms involve an additive character on the unipotent radical of $\bf H$, and we devised a twisted version of our theorem that yields necessary conditions for the existence of those mixed models. Our method of proof here is inspired by the theory of W-algebras. As an application we derive necessary conditions for the existence of Rankin-Selberg, Bessel, Klyachko and Shalika models. Our results are compatible with the recent Gan-Gross-Prasad conjectures for non-generic representations. We also prove more general results that ease the sphericity assumption on the subgroup, and apply them to local theta correspondence in type II and to degenerate Whittaker models.