Searching for charged Higgs bosons in top decays via the $t^*b$ channel

Abstract

Rare top-quark decays offer a sensitive probe of charged Higgs bosons with masses below the top mass, owing to the large $t\bar t$ production rate at the LHC and the distinctive final states involving leptons and $b$-jets. While existing searches target the $H^\pm\to \tau\nu$, $cs$, and $cb$ modes, the decay $H^\pm\to tb$ has been studied only for heavier charged Higgs bosons with an on-shell top quark in the final state. The low-mass off-shell channel $H^\pm\to t^*b$ therefore remains essentially unconstrained, even though it can become the dominant decay mode below the top-quark threshold owing to the large top Yukawa coupling. We study charged-Higgs production from the rare top decay $t\to H^\pm b$, followed by the decay $H^\pm\to t^*b$. Top-antitop production and decay hence give rise to a $t\bar t b\bar b$-like final state, which we constrain by reinterpreting recent ATLAS fiducial measurements in dileptonic events. We obtain model-independent limits of 1.9%-2.9% on the product of branching ratios ${\rm Br}(t\to H^\pm b)\times{\rm Br}(H^\pm\to t^*b)$ for charged-Higgs masses between 110 and 165 GeV, and interpret these bounds in several two-Higgs-doublet scenarios. While dedicated searches for conventional charged-Higgs decays dominate for canonical $Z_2$-symmetric models, the $t^*b$ reinterpretation becomes competitive near the top-quark threshold in the up-type aligned limit and provides relevant direct constraint in a top-philic scenario. In the top-specific Two-Higgs-Doublet Model, it additionally excludes a low-$\tan\beta$ region not covered by conventional searches. These results thus establish $t\bar t$ production with additional $b$-jets as a complementary probe of light charged Higgs bosons and nonstandard top-quark decays, and motivate dedicated analyses by the LHC collaborations.