Résumé : One of the main challenges in galaxy formation that has emerged recently is the early assembly of massive galaxies. The observed number density and the maximum stellar mass ($M_{\star}$) of massive galaxies in the early Universe appear to be higher than model predictions, which may pose a serious problem to the LCDM cosmology. A major limitation in many previous studies is the large uncertainty in estimating $M_{\star}$ due to the lack of constraints in the rest-frame near-infrared part of the spectral energy distribution, which is critical to determining $M_{\star}$ accurately. Here we use data from a large JWST/MIRI survey in the PRIMER program to carry out a systematic analysis of massive galaxies at $z \sim 3-8$, leveraging photometric constraints at rest-frame $\gtrsim 1 \mu$m. We find a significant reduction in the number and mass densities of massive galaxies at $z > 5$ compared to earlier results that did not use the MIRI photometry. Within the standard $\Lambda$CDM cosmology, our results require a moderate increase in the baryon-to-star conversion efficiency ($\epsilon$) towards higher redshifts and higher $M_{\star}$. For the most massive galaxies at $z\sim 8$, the required $\epsilon$ is $\sim 0.3$, in comparison to $\epsilon \sim 0.14$ for typical low-redshift galaxies. Our findings are consistent with models assuming suppressed stellar feedback due to the high gas density and the associated short free-fall time expected for massive halos at high redshift.