Chiral spin density wave order on frustrated honeycomb and bilayer triangle lattice Hubbard model at half-filling

Abstract

We study the Hubbard model on the frustrated honeycomb lattice with nearest-neighbor $t_1$ and second nearest-neighbor hopping $t_2$, which is isomorphic to the bilayer triangle lattice, using the SU(2)-invariant slave boson theory. We show that the Coulomb interaction $U$ induces antiferromagnetic (AF) chiral spin-density wave ($χ$-SDW) order in a wide range of $κ=t_2/t_1$ where both the two-sublattice AF order at small $κ$ and the decoupled three-sublattice 120$^\circ$ order at large $κ$ are strongly frustrated, leading to three distinct phases with different anomalous Hall responses. We find a continuous transition from a $χ$-SDW semimetal with anomalous Hall effect to a topological chiral Chern insulator exhibiting quantum anomalous Hall effect, followed by a discontinuous transition to a $χ$-SDW insulator with zero total Chern number but anomalous ac Hall effect.The $χ$-SDW is likely a generic phase of strongly correlated and highly frustrated hexagonal lattice electrons.

6 pages, 5 figures. To be published in Physical Review Letters