What is the physical meaning of the statement ''Lagrangian can only be defined up to a total derivative"? Considering an analogue for potential energy of a physical system, it can be unique up to an additive constant but this can be explained on the ground that we are really interested in the change of potential energy and this additive constant does not contribute to this.
 A: The physical meaning is that total derivative/divergence terms are just boundary terms in the action, and boundary conditions fixes the boundary, so they cannot enter actively into the stationary action principle nor alter the EL equations (assuming the variational problem is well-posed). See also this related Phys.SE post.
A: The action, essentially, is the product of energy (in a system) times the time (the evolution parameter). But energy has no meaning; only its differences. That is why action behaves like a potential, and is defined only modulo an additive constant (which should correspond to a boundary term emanating from a total time derivative). Thus, it is only the variation of an action which matters. 
A: Hamilton's principle is an statement about the action
$$
S = \int_{t_1}^{t_2} {\rm d}t ~L
$$
this means whatever dynamics you derive from this, it is based on the fact that $\delta S = 0$. With this in mind, imagine that for a Lagrangian $L$ you define a new functional of the form $L' = L + {\rm d}F/{\rm d}t$, so that the action becomes
$$
S' = \int_{t_1}^{t_2}{\rm d}t ~L' = \int_{t_1}^{t_2}{\rm d}t ~L + \int_{t_1}^{t_2}{\rm d}t ~\frac{{\rm d}F}{{\rm d}t} = S + [F(t_2) - F(t_1)]
$$
and clearly
$$
\delta S' = \delta S
$$
so the dynamics generated is the same. That is, that is, you can always change the Lagrangian by adding a derivative ant the dynamics is the same.
A: We use Lagrangians to construct equations of motion, and equations of motion to construct measurable predictions about how a system will evolve over time. So Lagrangians encode a kind of meta-pattern in the prediction construction process, and do so quite efficiently with a lot of flexibility in how the system is described. If two Lagrangians produce the same equations of motion in all cases, then the difference between them is a distinction without a difference. Similar, if more complicated, to the statement $A + B = B + A$.
