The FENIKS Survey: Spectroscopic Confirmation of Massive Quiescent Galaxies at z ~ 3-5

Abstract: The measured ages of massive, quiescent galaxies at $z\sim 3-4$ imply that massive galaxies quench as early as $z\sim 6$. While the number of spectroscopic confirmations of quiescent galaxies at $z < 3$ has increased over the years, there are only a handful at $z > 3.5$. We report spectroscopic redshifts of one secure ($z=3.757$) and two tentative ($z = 3.336$, $z=4.673$) massive ($\log(M_\ast/M_\odot) > 10.3$) quiescent galaxies with 11 hours of Keck/MOSFIRE $K$-band observations. Our candidates were selected from the FENIKS survey, which uses deep Gemini/Flamingos-2 $K_b$ $K_r$ imaging optimized for increased sensitivity to the characteristic red colors of galaxies at $z > 3$ with strong Balmer/4000 \AA\ breaks. The rest-frame $UVJ$ and $(ugi)_s$ colors of 3/4 quiescent candidates are consistent with $1-2$ Gyr old stellar populations. This places these galaxies as the oldest objects at these redshifts, and challenges the notion that quiescent galaxies at $z > 3$ are all recently-quenched, "post-starburst'' galaxies. Our spectroscopy shows that the other quiescent-galaxy candidate is a broad-line AGN ($z = 3.594$) with strong, redshifted $H\beta$+[O III] emission with a velocity offset $>1000$ km/s, indicative of a powerful outflow. The star-formation history of our highest redshift candidate suggests that its progenitor was already in place by $z \sim 7-11$, reaching $\sim$ 10$^{11} M_{\odot}$ by $z \simeq 10$. These observations reveal the limit of what is possible with deep near-infrared photometry and targeted spectroscopy from the ground and demonstrate that secure spectroscopic confirmation of quiescent galaxies at $z > 4$ is only feasible with JWST.