Supermassive black holes are growing slowly by $z\sim5$
We investigate the black hole mass function at $z\sim5$ using XQz5, our recent sample of the most luminous quasars between the redshifts $4.5 < z < 5.3$. We include 72 quasars with black hole masses estimated from velocity-broadened emission-line measurements and single-epoch virial prescriptions in the footprint of a highly complete parent survey. The sample mean Eddington ratio and standard deviation is $\logλ\approx -0.20\pm0.24$. The completeness-corrected mass function is modelled as a double power-law, and we constrain its evolution across redshift assuming accretion-dominated mass growth. We estimate the evolution of the mass function from $z=5-4$, presenting joint constraints on accretion properties through a measured dimensionless e-folding parameter, $k_{\rm{ef}} \equiv \langleλ\rangle U (1-ε)/ε= 1.79\pm0.06$, where $\langleλ\rangle$ is the mean Eddington ratio, $U$ is the duty cycle, and $ε$ is the radiative efficiency. If these supermassive black holes were to form from seeds smaller than $10^8\,M_{\odot}$, the growth rate must have been considerably faster at $z\gg5$ than observed from $z=5-4$. A growth rate exceeding $3\times$ the observed rate would reduce the initial heavy seed mass to $10^{5-6}\,M_{\odot}$, aligning with supermassive star and/or direct collapse seed masses. Stellar mass ($10^2\,M_{\odot}$) black hole seeds would require $\gtrsim4.5\times$ the observed growth rate at $z\gg5$ to reproduce the measured active black hole mass function. A possible pathway to produce the most extreme quasars is radiatively inefficient accretion flow, suggesting black holes with low angular momentum or photon trapping in supercritically accreting thick discs.