Chiral spin state and nematic ferromagnet in the spin-1 Kitaev-$\Gamma$ model

Abstract

The higher-spin Kitaev magnets, in which the Kitaev interaction and off-diagonal exchange couplings are overwhelmingly large, have emerged as a fertile avenue to explore exotic phases and unusual excitations. In this work, we study the quantum phase diagram of the spin-1 Kitaev-$\Gamma$ model on the honeycomb lattice using density-matrix renormalization group. It harbours six distinct phases and the intriguing findings are three magnetically ordered phases in which both time-reversal symmetry and lattice symmetry albeit of different sort are broken spontaneously. The chiral spin state originates from the order-by-disorder effect and exhibits an almost saturated scalar spin chirality at the quantum level. Depending on the relative strength of the two interactions, it also features columnar-like or plaquette-like dimer pattern as a consequence of the translational symmetry breaking. In parallel, the nematic ferromagnets are situated at ferromagnetic Kitaev side and possess small but finite ferromagnetic ordering. The lattice-rotational symmetry breaking enforces nonequivalent bond energy along one of the three bonds. Although the intrinsic difference between the two nematic ferromagnets remains elusive, the discontinuities in the von Neumann entropy, hexagonal plaquette operator, and Wilson loop operator convincingly suggest that they are separated via a first-order phase transition.