The “memory effect” is the permanent relative displacement of test particles after the passage of gravitational radiation. It is associated with both the propagation of massive bodies out to timelike infinity (“ordinary memory”) or the propagation of radiation out to null infinity (“null memory”). The memory effect can be characterized by the failure of the shear tensor at order $1/r$ to return to zero at late times, even though it is “pure gauge.” Closely analogous effects occur in electromagnetism, where the vector potential at order $1/r$ fails to return to zero even though it is “pure gauge.” In both cases, the Fourier transform of the radiative field has divergent behavior at low frequencies. This gives rise to infrared divergences (i.e., infinite numbers of “soft” gravitons/photons) in the quantum field theory description if one attempts to describe these states as vectors in the usual Fock Hilbert space representation. To obtain a mathematically sensible quantum scattering theory, one must allow states with nonvanishing memory in the “in” and “out” Hilbert spaces. An elegant solution to this problem in massive quantum electrodynamics was given by Kulish and Fadeev, who constructed a Hilbert space of incoming/outgoing charged particle states that are “dressed” with radiative fields of corresponding memory, so as to yield vanishing large gauge charges at spatial infinity. However, we show that this type of construction fails in quantum gravity. The primary underlying reason is that the “dressing” contributes to null memory, thereby invalidating the construction of eigenstates of large gauge charges. In quantum gravity, there does not appear to be any choice of (separable) Hilbert space of incoming/outgoing states that can accommodate all scattering states. Thus, we argue that scattering should be described at the level of algebraic incoming/outgoing states rather than attempting to artificially restrict states to a particular Hilbert space.