Degenerate principal series representations and their holomorphic extensions

Abstract

Let $X=H/L$ be an irreducible real bounded symmetric domain realized as a real form in an Hermitian symmetric domain $D=G/K$. The intersection $S$ of the Shilov boundary of $D$ with $X$ defines a distinguished subset of the topological boundary of $X$ and is invariant under $H$ and can also be realized as $S=H/P$ for certain parabolic subgroup $P$ of $H$. We study the spherical representations $Ind_P^H(\lam)$ of $H$ induced from $P$. We find formulas for the spherical functions in terms of the Macdonald ${}_2F_1$ hypergeometric function. This generalizes the earlier result of Faraut-Koranyi for Hermitian symmetric spaces $D$. We consider a class of $H$-invariant integral intertwining operators from the representations $Ind_P^H(\lam)$ on $L^2(S)$ to the holomorphic representations of $G$ on $D$ restricted to $H$. We construct a new class of complementary series for the groups $H=SO(n, m)$, $SU(n, m)$ (with $n-m >2$) and $Sp(n, m)$ (with $n-m>1$). We realize them as a discrete component in the branching rule of the analytic continuation of the holomorphic discrete series of $G=SU(n, m)$, $SU(n, m)\times SU(n, m)$ and $SU(2n, 2m)$ respectively.