Moduli of non-standard Nikulin surfaces in low genus

Abstract

Primitively polarized genus $g$ Nikulin surfaces $(S,M,H)$ are of two types, that we call standard and non-standard depending on whether the lattice embedding $\mathbb{Z}[H] \oplus_{\perp} \mathbf{N} \subset \rm{Pic}(S)$ is primitive. Here $H$ is the genus $g$ polarization and $\mathbf{N}$ is the Nikulin lattice. We concentrate on the non-standard case, which only occurs in odd genus. In particular, we study the birational geometry of the moduli space of non-standard Nikulin surfaces of genus $g$ and prove its rationality for $g=7,11$ and the existence of a rational double cover of it when $g=9$. Furthermore, if $(S,M,H)$ is general in the above moduli space and $(C,M|_C)$ is a general Prym curve in $|H|$, we determine the dimension of the family of non-standard Nikulin surfaces of genus $g$ containing $(C, M|_C)$ for $3\leq g\leq 11$; this completes the study of the Prym-Nikulin map initiated in our previous work.