Have we seen all the galaxies that comprise the cosmic infrared background at 250\,$μ$m $\le λ\le$ 500\,$μ$m?
The cosmic infrared background (CIB) provides a fundamental observational constraint on the star-formation history of galaxies over cosmic history. We estimate the contribution to the CIB from catalogued galaxies in the COSMOS field by using a novel map fitting technique on the \textit{Herschel} SPIRE maps. Prior galaxy positions are obtained using detections over a large range in wavelengths in the $K_{\rm s}$--3\,GHz range. Our method simultaneously fits the galaxies, the system foreground, and the leakage of flux from galaxies located in masked areas and corrects for an "over-fitting" effect not previously accounted for in stacking methods. We explore the contribution to the CIB as a function of galaxy survey wavelength and depth. We find high contributions to the CIB with the deep $r$ ($m_{\rm AB} \le 26.5$), $K_{\rm s}$ ($m_{\rm AB} \le 24.0$) and 3.6\,$μ$m ($m_{\rm AB} \le 25.5$) catalogues. We combine these three deep catalogues and find a total CIB contributions of 10.5 $\pm$ 1.6, 6.7 $\pm$ 1.5 and 3.1 $\pm$ 0.7\,nWm$^{-2}$sr$^{-1}$ at 250, 350 and 500\,$μ$m, respectively. Our CIB estimates are consistent with recent phenomenological models, prior based SPIRE number counts and with (though more precise than) the diffuse total measured by FIRAS. Our results raise the interesting prospect that the CIB contribution at $λ\le 500\,μ$m from known galaxies has converged. Future large-area surveys like those with the Large Synoptic Survey Telescope are therefore likely to resolve a substantial fraction of the population responsible for the CIB at 250\,$μ$m $\leq λ\leq$ 500\,$μ$m.