When Could Abelian Fractional Topological Insulators Exist in Twisted MoTe$_2$ (and Other Systems)

Abstract

Using comprehensive exact diagonalization calculations on $\theta \approx 3.7 ^{\circ}$ twisted bilayer MoTe$_2$ ($t$MoTe$_2$), as well as idealized Landau level models also relevant for lower $\theta$, we extract general principles for engineering fractional topological insulators (FTIs) in realistic situations. First, in a Landau level setup at $\nu=1/3+1/3$, we investigate what features of the interaction destroy an FTI. For both pseudopotential interactions and realistic screened Coulomb interactions, we find that sufficient suppression of the short-range repulsion is needed for stabilizing an FTI. We then study $\theta \approx 3.7 ^{\circ}$ $t$MoTe$_2$ with realistic band-mixing and anisotropic non-local dielectric screening. Our finite-size calculations only find an FTI phase at $\nu=-4/3$ in the presence of a significant additional short-range attraction $g$ that acts to counter the Coulomb repulsion at short distances. We discuss how further finite-size drifts, dielectric engineering, Landau level character, and band-mixing effects may reduce the required value of $g$ closer towards the experimentally relevant conditions of $t$MoTe$_2$. Projective calculations into the $n=1$ Landau level, which resembles the second valence band of $\theta\simeq 2.1^\circ$ $t$MoTe$_2$, do not yield FTIs for any $g$, suggesting that FTIs at low-angle $t$MoTe$_2$ for $\nu=-8/3$ and $-10/3$ may be unlikely. While our study highlights the challenges, at least for the fillings considered, to obtaining an FTI with transport plateaus, even in large-angle $t$MoTe$_2$ where fractional Chern insulators are experimentally established, we also provide potential sample-engineering routes to improve the stability of FTI phases.