Generalized Whittaker quotients of Schwartz functions on G-spaces

Abstract

Let $G$ be a reductive group over a local field $F$ of characteristic zero, Archimedean or not. Let $X$ be a $G$-space. In this paper we study the existence of generalized Whittaker quotients for the space of Schwartz functions on $X$, considered as a representation of $G$. We show that the set of nilpotent elements of the dual space to the Lie algebra such that the corresponding generalized Whittaker quotient does not vanish contains the nilpotent part of the image of the moment map, and lies in the closure of this image. This generalizes recent results of Prasad and Sakellaridis. Applying our theorems to symmetric pairs $(G,H)$ we show that there exists an infinite-dimensional $H$-distinguished representation of $G$ if and only if the real reductive group corresponding to the pair $(G,H)$ is non-compact. For quasi-split $G$ we also extend to the Archimedean case the theorem of Prasad stating that there exists a generic $H$-distinguished representation of $G$ if and only if the real reductive group corresponding to the pair $(G,H)$ is quasi-split. In the non-Archimedean case our result also gives rather sharp bounds on the wave-front sets of distinguished representations. The results in the present paper can be used to recover many of the vanishing results on periods of automorphic forms proved by Ash-Ginzburg-Rallis. This follows from our Corollary H when combined with the restrictions on the Whittaker support of cuspidal automorphic representations proven in [GGS21].