I always thought the uniformity in the temperature of the CMB was supposed to be expected, since it's a much more probable initial condition for the universe. I finally found someone explaining what I mean in much better words (link):
Horizon problem isn't really a problem
If we examine from statistical mechanics principles what thermal equilibrium really means, we see that it is the most probable macrostate for a system (in other words, the state with highest entropy). Systems evolve towards thermal equilibrium not because nature has any sort of preference for evening out energy among all degrees of freedom, but simply because having a roughly equal partition of energy among degrees of freedom is OVERWHELMINGLY probable.
For exactly the same reason why it is overwhelmingly probable for a closed system to move toward thermal equilibrium, it is overwhelmingly probable for a completely randomly selected initial condition to be in thermal equilibrium. No causal contact is necessary.
The only "counter-argument" I could find for that, ironically enough, comes from Jason Lisle (link):
(...) in the early universe, the temperature of the CMB would have been very different at different places in space due to the random nature of the initial conditions.
But if that "random nature of the initial conditions" is of the same order of magnitude as quantum fluctuations, wouldn't that apply to the early instants of inflation too? If so, how would thermal equilibrium be even possible under such quantum fluctuations during inflation?