Coverage is not enough: Frequentist tests of simulation-based inference for primordial non-Gaussianity

Abstract: (Abridged) Simulation-based inference (SBI) has emerged as a powerful framework for extracting cosmological information from complex, non-linear data where analytical likelihoods are unavailable. Its reliability is commonly assessed using coverage-based diagnostics under the prior predictive distribution, which probe calibration only in an averaged sense and do not constrain posterior behavior at fixed parameter value, the regime relevant for practical inference. We investigate these limitations in the context of primordial non-Gaussianity, parameterized by $f_\mathrm{NL}$, using simulations of the dark matter halo field. We compare SBI based on contrastive neural ratio estimation (CNRE) with likelihood-based inference (LBI) using the power spectrum, bispectrum, and wavelet scattering transform (WST) coefficients across 1000 realizations. SBI and LBI agree well on posterior means and skewness, while the variance agrees on average but shows weaker realization-by-realization consistency. Larger differences arise in the kurtosis, indicating discrepancies in the posterior tails. These effects are already present for the power spectrum - where the Gaussian likelihood assumed in LBI is best justified - and are most pronounced for the combined power spectrum and bispectrum, where SBI posteriors are often underconfident and can yield weaker constraints than either statistic individually, despite passing coverage tests. WST coefficients further tighten constraints on $f_\mathrm{NL}$, even when restricted to large scales. Our results highlight both the potential of higher-order statistics and the need for validation strategies that probe the posterior shape beyond standard coverage diagnostics.