Résumé : Cosmic rays (CRs) with energies $\ll$ TeV comprise a significant component of the interstellar medium (ISM). Major uncertainties in CR behavior on observable scales (much larger than CR gyroradii) stem from how magnetic fluctuations scatter CRs in pitch angle. Traditional first-principles models, which assume these magnetic fluctuations are weak and uniformly scatter CRs in a homogeneous ISM, struggle to reproduce basic observables such as the dependence of CR residence times and scattering rates on rigidity. We therefore explore a new category of "patchy" CR scattering models, wherein CRs are predominantly scattered by intermittent strong scattering structures with small volume-filling factors. These models produce the observed rigidity dependence with a simple size distribution constraint, such that larger scattering structures are rarer but can scatter a wider range of CR energies. To reproduce the empirically-inferred CR scattering rates, the mean free path between scattering structures must be $\ell_{\rm mfp} \sim 10$ pc at GeV energies. We derive constraints on the sizes, internal properties, mass/volume-filling factors, and the number density any such structures would need to be both physically and observationally consistent. We consider a range of candidate structures, both large-scale (e.g. H II regions) and small-scale (e.g. intermittent turbulent structures, perhaps even associated with radio plasma scattering) and show that while many macroscopic candidates can be immediately ruled out as the primary CR scattering sites, many smaller structures remain viable and merit further theoretical study. We discuss future observational constraints that could test these models.