The Observable Universe is generally said to contain all space that could "in principle" have had a causal impact on Earth, but the exact limits of the "in principle" causal interaction go unspecified. Wikipedia notes some astrophysicists distinguish the Visible Universe, all space that was in our past light cone at recombination, from a broader Observable Universe, all space that was in our past light cone at the end of the inflationary epoch. While obviously the first definition has more practical importance in cosmology, the latter seems to be much truer to the meaning of "in principle".
Is this latter definition known to be "final"? In other words, are there theoretical reasons to believe that there actually was no causal influence between the OU and its neighboring regions of space during the inflationary epoch, or that any such interactions had no causal impact on post-inflationary dynamics? If so, what are they?
Intuitively, it seems like if you define $OU(t)$ as the Earth-centered ball of present-day space that was in our past light cone $t$ seconds after the Big Bang, the size of this ball grows without limit as $t$ approaches 0. This implies the entire Universe, even if it is infinitely large, is "in principle" causally connected to Earth. At what point does this intuition go wrong?