Abstract: We present the results of a systematic study of the rest-frame optical morphology of quiescent galaxies at $z \geq 3$ using the Near-Infrared Camera (NIRCam) onboard $JWST$. Based on a sample selected by $UVJ$ color or $NUVUVJ$ color, we focus on 26 quiescent galaxies with $9.8<\log{(M_\star/M_\odot)}<11.4$ at $2.8<z_{\rm phot}<4.6$ with publicly available $JWST$ data. Their sizes are constrained by fitting the S\'ersic profile to all available NIRCam images. We see a negative correlation between the observed wavelength and the size in our sample and derive their size at the rest-frame $0.5\, {\rm \mu m}$ taking into account this trend. Our quiescent galaxies show a significant correlation between the rest-frame $0.5\, {\rm \mu m}$ size and the stellar mass at $z\geq3$. The analytical fit for them at $\log{(M_\star/M_\odot)}>10.3$ implies that our size - stellar mass relations are below those at lower redshifts, with the amplitude of $\sim0.6\, {\rm kpc}$ at $M_\star = 5\times 10^{10}\, M_\odot$. This value agrees with the extrapolation from the size evolution of quiescent galaxies at $z<3$ in the literature, implying that the size of quiescent galaxies increases monotonically from $z\sim3-5$. Our sample is mainly composed of galaxies with bulge-like structures according to their median S\'ersic index and axis ratio of $n\sim3-4$ and $q\sim0.6-0.8$, respectively. On the other hand, there is a trend of increasing fraction of galaxies with low S\'ersic index, suggesting $3<z<5$ might be the epoch of onset of morphological transformation with a fraction of very notable disky quenched galaxies.