I am reading Griffiths' Introduction to Electrodynamics in which he shows that the retarded and advanced potentials, e.g. the retarded scalar potential
$$ V(\mathbf{r},t) = \frac{1}{4\pi\epsilon_0} \int \frac{\rho(\mathbf{r}',t_r)}{\lvert \mathbf{r} - \mathbf{r}' \rvert} \mathrm{d}\tau', \quad t_r \equiv t - \frac{\lvert \mathbf{r} - \mathbf{r}' \rvert}{c},$$
are solutions to the inhomogeneous wave equation and satisfy the Lorenz condition. He then rejects the advanced potentials by invoking the principle of causality, stating that it is not unreasonable to believe that electromagnetic influences propagate forward and not backward in time.
I can only imagine that the reasonableness of this belief is rooted in our naive experience of the world as time-asymmetric. However, the time asymmetry that we experience on a daily basis are (at least usually) not due to a fundamental asymmetry in the fundamental laws of physics, like Maxwell's equations (insofar as a classical theory can be fundamental, but to my knowledge QED is also time symmetric), but to the emergent second law of thermodynamics. It is not clear to me that we can use our intuition about macroscopic phenomena to reason about microscopic phenomena, and even if we could, the second law is only probabilistic and would not allow us to reject forward propagation of electromagnetic influences outright and declare them impossible, only conclude that they are unlikely.
Furthermore, I assume that experiments have verified the retarded and not the advanced potentials? If so, then why invoke the principle of causation at all? And more importantly, would this not show a time asymmetry in the laws of classical electrodynamics? I suppose the principle could still considered external to the theory (like the homogeneity and isotropy of space), but at least the application to this particular case would have been shown experimentally to be justified.
So what exactly is it that allows us to apply the principle of causality?