Nematic orders and nematicity-driven topological phase transition in FeSe

Abstract

We investigate nematic states in both bulk FeSe and FeSe thin films. It is found that their band structures and signature features that were observed in a variety of experiments can be perfectly explained by introducing the $d$-wave nematic orders that are required to have contributions from all $t_{2g}$ d-orbitals, which contradicts the conventional wisdom that the nematicity is simply driven by the orbital degeneracy between the $d_{xz}$ and $d_{yz}$ orbitals. These orders can be generated by the Coulomb interaction between the nearest neighbor Fe sites. In the presence of spin-orbital couplings, we predict that the nematic order can drive a topological quantum phase transition through a band inversion at the $M$ point of the Brillouin zone to produce topologically protected edge states near the Fermi level. The prediction makes FeSe as a tunable system to integrate topological properties into high temperature superconductivity to realize Majorana related physics.