Résumé : The Sun is embedded in the so-called Local Bubble (LB) -- a cavity of hot plasma created by supernova explosions and surrounded by a shell of cold, dusty gas. Knowing the local distortion of the Galactic magnetic field associated with the LB is critical for the modeling of interstellar polarization data at high Galactic latitudes. In this his paper, we relate the structure of the Galactic magnetic field on the LB scale to three-dimensional (3D) maps of the local interstellar medium (ISM). First, we extracted the geometry of the LB shell, its inner surface, in particular from 3D dust extinction maps of the local ISM. We expanded the shell inner surface in spherical harmonics, up to a variable maximum multipole degree, which enabled us to control the level of complexity for the modeled surface. Next, we applied an analytical model for the ordered magnetic field in the shell to the modeled shell surface. This magnetic field model was successfully fitted to the \textit{Planck} 353~GHz dust polarized emission maps over the Galactic polar caps. For each polar cap, the direction of the mean magnetic field derived from dust polarization (together with the prior that the field points toward longitude $90^\circ \pm 90^\circ$) is found to be consistent with the Faraday spectra of the nearby diffuse synchrotron emission. Our work presents a new approach to modeling the local structure of the Galactic magnetic field. We expect our methodology and our results to be useful both in modeling the local ISM as traced by its different components and in modeling the dust polarized emission, which is a long-awaited input for studies of the polarized foregrounds for cosmic microwave background.