Annihilator varieties, highest derivatives, Whittaker functionals, and rank for unitary representations of GL(n,R)

Abstract

In this paper we study irreducible unitary representations of GL(n,R) and prove a number of results. Our first result establishes a precise connection between the annihilator of a representation and the existence of degenerate Whittaker functionals, for both smooth and K-finite vectors, thereby generalizing results of Kostant, Matumoto and others. Our second result relates the annihilator to the sequence of highest derivatives, as defined in this setting by one of the authors. Based on those results, we suggest a new notion of rank of a smooth admissible representation of GL(n,R), which for unitarizable representations refines Howe's notion of rank. Our third result computes the highest derivatives for (almost) all unitary representations in terms of the Vogan classification. We also indicate briefly the analogous results over complex and p-adic fields.