Competing ferromagnetic and antiferromagnetic phases on the frustrated Ising honeycomb lattice

Abstract

We investigate the frustrated $J_1$-$J_2$-$J_3$ Ising model on the honeycomb lattice, featuring first- and second-neighbor ferromagnetic couplings ($J_1>0$ and $J_2>0$) and third-neighbor antiferromagnetic interactions ($J_3<0$). Using the cluster mean-field method, we analyze the phase transitions in the regime $1/2 < J_2/J_1 \le 1$, where ferromagnetic and antiferromagnetic phases compete. Our results reveal that near the strongly frustrated limit $J_3/J_1 = -1$, the system exhibits order-by-disorder state selection, tricritical and bicritical behavior, critical endpoints, and two successive phase transitions. The ferromagnetic-paramagnetic transition remains second order across the entire interaction range, whereas the antiferromagnetic-paramagnetic boundary shows a richer behavior, including both first- and second-order transitions as well as tricriticality. Increasing the second-neighbor coupling $J_2/J_1$ narrows the range of $J_3/J_1$ where first-order antiferromagnetic-paramagnetic transitions occur; beyond a certain threshold, only second-order order-disorder transitions persist. Consequently, the tricritical point shifts toward $J_3/J_1 \approx -1$ as $J_2/J_1$ increases, culminating in a bicritical point where the antiferromagnetic, ferromagnetic, and paramagnetic phases meet.