Orbital-selective Mott phase and spin nematicity in Ni-substituted FeTe$_{0.65}$Se$_{0.35}$ single crystals

Abstract

The normal state in iron chalcogenides is metallic but highly unusual, with orbital and spin degrees of freedom partially itinerant or localized depending on temperature, leading to many unusual features. In this work, we report on the observations of two of such features, the orbital selective Mott phase (OSMP) and spin nematicity, evidenced in magnetization and magnetotransport [resistivity, Hall effect, anisotropic magnetoresistance (AMR)] of Fe$_{1-y}$Ni$_y$Te$_{0.65}$Se$_{0.35}$ single crystals, with $0 < y < 0.21$. Substitution of Ni dopes crystals with electrons, what eliminates some of the hole pockets from Fermi level, leaving only one, originating from $d_{xy}$ orbital. This leads to electron-dominated conduction at low $T$ for $y \gtrsim 0.06$. However, at high temperatures, $T \gtrsim 125 \div 178$ K, the conduction reverses to hole-dominated. Anomalies in magnetization and resistivity are observed at temperatures which approach high-$T$ boundary of the electron-dominated region. Analysis of these effects suggests a link with the appearance of the $d_{z^2}$ hole pockets at X points of the Brillouin zone in the OSMP phase, facilitated by the localization of $d_{xy}$ orbital, as recently reported by angular resolved photoemission experiments ($\textit{J. Huang et al., Commun. Phys. 5, 29 (2022)}$). The low-$T$ AMR shows mixed 4-fold and 2-fold rotational symmetry of in-plane magnetocrystalline anisotropy, with the 4-fold term the largest at small $y$, and suppressed at intermediate $y$. These results are consistent with the mixed stripe/bicollinear magnetic correlations at small $y$, and suppression of stripe correlations at intermediate $y$, indicating development of spin nematicity with increasing Ni doping, which possibly contributes to the suppression of superconductivity.