Fossil groups origins II. Unveiling the formation of the brightest group galaxies through their scaling relations

Abstract

(Abridged) Fossil systems are galaxy associations dominated by a relatively isolated, bright elliptical galaxy, surrounded by a group of smaller galaxies lacking L* objects. We analyzed the near-infrared photometric and structural properties of a sample of 20 BGGs present in FGs in order to better understand their formation mechanisms. Their surface-brightness distribution was fitted to a Sersic profile using the GASP2D algorithm. Then, the standard scaling relations were derived for the first time for these galaxies and compared with those of normal ellipticals and brightest cluster galaxies in non-fossil systems. The BGGs presented in this study represent a subset of the most massive galaxies in the Universe. We found that their ellipticity profiles are continuously increasing with the galactocentric radius. Our fossil BCGs follow closely the fundamental plane described by normal ellipticals. However, they depart from both the log \sigma_0 vs. log L_{K_{s}} and log r_{\rm e} vs. log L_{K_{s}} relations described by intermediate mass ellipticals. This occurs in the sense that our BGGs have larger effective radii and smaller velocity dispersions than those predicted by these relations. We also found that more elliptical galaxies systematically deviate from the previous relations while more rounder object do not. No similar correlation was found with the Sersic index. The derived scaling relations can be interpreted in terms of the formation scenario of the BGGs. Because our BGGs follow the fundamental plane tilt but they have larger effective radii than expected for intermediate mass ellipticals, we suggest that they only went through dissipational mergers in a early stage of their evolution and then assembled the bulk of their mass through subsequent dry mergers, contrary to previous claims that BGGs in FGs were formed mainly by the merging of gas-rich galaxies.