Résumé : (Abridged) Context. To directly image rocky exoplanets in reflected (polarized) light, future space- and ground-based high-contrast imagers and telescopes aim to reach extreme contrasts at close separations from the star. However, the achievable contrast will be limited by reflection-induced polarization aberrations. While polarization aberrations can be modeled numerically, such computations provide little insight into the full range of effects, their origin and characteristics, and possible ways to mitigate them. Aims. We aim to understand polarization aberrations produced by reflection off flat metallic mirrors at the fundamental level. Methods. We used polarization ray tracing to numerically compute polarization aberrations and interpret the results in terms of the polarization-dependent spatial and angular Goos-H\"anchen and Imbert-Federov shifts of the beam of light as described with closed-form mathematical expressions in the physics literature. Results. We find that all four beam shifts are fully reproduced by polarization ray tracing and study the origin, characteristics, sizes, and directions of the shifts. Of the four beam shifts, only the spatial Goos-H\"anchen and Imbert-Federov shifts are relevant for high-contrast imagers and telescopes because these shifts are visible in the focal plane and create a polarization structure in the PSF that reduces the performance of coronagraphs and the polarimetric speckle suppression close to the star. Conclusions. The beam shifts in an optical system can be mitigated by keeping the f-numbers large and angles of incidence small. Most importantly, mirror coatings should not be optimized for maximum reflectivity, but should be designed to have a retardance close to 180{\deg}. The insights from our study can be applied to improve the performance of current and future high-contrast imagers, especially those in space and on the ELTs.