Résumé : Any non-trivial scattering with any massless fields in four spacetime dimensions will generically produce an "out" state with memory which gives rise to infrared divergences in the standard $S$-matrix. To obtain an infrared-finite scattering theory, one must suitably include states with memory. However, except in the case of QED with massive charged particles, asymptotic states with memory that have finite energy and angular momentum have not been constructed for more general theories (e.g. massless QED, Yang-Mills and quantum gravity). To this end, we construct direct-integral representations over the "Lorentz orbit" of a given memory and classify all "orbit space representations" that have well-defined energy and angular momentum. We thereby provide an explicit construction of a large supply of physical states with memory as well as the explicit action of the BMS charges all states. The construction of such states is a key step toward the formulation of an infrared-finite scattering theory. While we primarily focus on the quantum gravitational case, we outline how the methods presented in this paper can be applied to obtain representations of the Poincar\'e group with memory for more general quantum field theories.