Spectra for Gelfand pairs associated with the free two step nilpotent lie group

Abstract

Let $F(n)$ be a connected and simply connected free 2-step nilpotent lie group and $K$ be a compact subgroup of Aut($F(n)$). We say that $(K,F(n))$ is a Gelfand pair when the set of integrable $K$-invariant functions on $F(n)$ forms an abelian algebra under convolution. In this paper, we consider the case when $K=O(n)$. In [1], we know the only possible Galfand pairs for $(K,F(n))$ is $(O(n),F(n))$, $(SO(n),F(n))$. So we just consider the case $(O(n),F(n))$, the other case can be obtained in the similar way.We study the natural topology on $\Delta (O(n),F(n))$ given by uniform convergence on compact subsets in $F(n)$. We show $\Delta (O(n),F(n))$ is a complete metric space. Our main result gives a necessary and sufficient result for the sequence of the "type 1" bounded $O(n)$-spherical functions uniform convergence to the "type 1" bounded $O(n)$-spherical function on compact sets in $F(n)$. What's more, the "type 1" bounded $O(n)$-spherical functions are dense in $\Delta (O(n),F(n))$. Further, we define the Fourier transform according to the "type 2" bounded $O(n)$-spherical functions and gives some basic properties of it.