Hecke triangle groups, transfer operators and Hausdorff dimension

Abstract

We consider the family of Hecke triangle groups $ \Gamma_{w} = \langle S, T_w\rangle $ generated by the M\"obius transformations $ S : z\mapsto -1/z $ and $ T_{w} : z \mapsto z+w $ with $ w > 2.$ In this case the corresponding hyperbolic quotient $ \Gamma_{w}\backslash\mathbb{H}^2 $ is an infinite-area orbifold. Moreover, the limit set of $ \Gamma_w $ is a Cantor-like fractal whose Hausdorff dimension we denote by $ \delta(w). $ The first result of this paper asserts that the twisted Selberg zeta function $ Z_{\Gamma_{ w}}(s, \rho) $, where $ \rho : \Gamma_{w} \to \mathrm{U}(V) $ is an arbitrary finite-dimensional unitary representation, can be realized as the Fredholm determinant of a Mayer-type transfer operator. This result has a number of applications. We study the distribution of the zeros in the half-plane $\mathrm{Re}(s) > \frac{1}{2}$ of the Selberg zeta function of a special family of subgroups $( \Gamma_w^n )_{n\in \mathbb{N}} $ of $\Gamma_w$. These zeros correspond to the eigenvalues of the Laplacian on the associated hyperbolic surfaces $X_w^n = \Gamma_w^n \backslash \mathbb{H}^2$. We show that the classical Selberg zeta function $Z_{\Gamma_w}(s)$ can be approximated by determinants of finite matrices whose entries are explicitly given in terms of the Riemann zeta function. Moreover, we prove an asymptotic expansion for the Hausdorff dimension $\delta(w)$ as $w\to \infty$.