How lonely are the Binary Compact Objects Detected by the LIGO-Virgo-KAGRA Collaboration?
Gravitational-wave (GW) observations of compact binary coalescences (CBCs) are traditionally interpreted under the assumption that the binary evolves in isolation. However, in realistic astrophysical environments, brief three-body encounters may perturb the binary's orbital evolution and imprint deviations on the emitted GWs. We develop a physically motivated model for such interactions, retaining Newtonian three-body dynamics supplemented by leading-order ($2.5$PN) radiation-reaction within the binary. We show that such encounters produce a distinctive morphology of dephasing and amplitude modulation in GWs. We search for this kind of distortion from the LIGO--Virgo--KAGRA (LVK) GW catalog GWTC-4 on three events: GW170817, GW190814, and GW230627\_015337, chosen based on high SNR and in-band duration $\gtrsim 10~\mathrm{s}$. We find no statistically significant deviation in the data, which translates into constraints on the absence of any intermediate-mass black hole in the mass range above $\sim 10^2$ M$_\odot$ in the vicinity of these binaries of radius approximately $10^{-1}~\mathrm{AU}$. This arises from robust exclusions arising from fly-by interactions that would dynamically disrupt the binary and are directly ruled out independent of waveform modelling, placing the first upper bound on intermediate-mass black holes near these GW events. In future, with the availability of long-duration GW signals, this new avenue can probe encounters of the binary GW sources with compact objects of lighter masses at distances farther away than 1 AU and hence opens a new window to probe the population of individual compact objects of both astrophysical and primordial origin in astrophysical systems of dense environments ranging from galactic centers to dense globular clusters.