Odd and even Maass cusp forms for Hecke triangle groups, and the billiard flow

Abstract

By a transfer operator approach to Maass cusp forms and the Selberg zeta function for cofinite Hecke triangle groups, M. M\"oller and the author found a factorization of the Selberg zeta function into a product of Fredholm determinants of transfer-operator-like families: $Z(s) = \det(1-\mc L_s^+)\det(1-\mc L_s^-)$. In this article we show that the operator families $\mc L_s^\pm$ arise as families of transfer operators for the triangle groups underlying the Hecke triangle groups, and that for $s\in\C$, $\Rea s=\tfrac12$, the operator $\mc L_s^+$ (resp. $\mc L_s^-$) has a 1-eigenfunction if and only if there exists an even (resp. odd) Maass cusp form with eigenvalue $s(1-s)$. For nonarithmetic Hecke triangle groups, this result provides a new formulation of the Phillips-Sarnak conjecture on nonexistence of even Maass cusp forms.