Why does the "gravitational potential" - something defined by convention - rule time dilation and length contraction in GR? Assume that we have an exactly uniform gravitational field like that occur for an infinitely large plate, yet with a finite mass. As we know, two similar clocks located in a specific alignment in the field with different distances away from the plate, and at rest WRT the plate, undergo similar gravity, and thus the clocks are expected to run at the same rate. Even all the experiments performed in the compartments within which the clocks are located have the same outcomes.
However, according to GR, the clock which is nearer to the plate runs slower as viewed by the other clock located farther from the plate just because the nearer clock is in a lower (more negative) gravitational potential regardless of the strength of the gravitational field. Why this the case?
If the gravitational potential is something determined by convention, why and how it has become so important, rather than acceleration with real physical impacts, in affecting some real phenomena such as time dilation and length contraction? As far as I think, those physical qualities determined by convention are somehow apparent. Therefore, I think it is as if we claim that because the apparent size of the farther clock is smaller, thus this apparent phenomenon affects time rates or the length measurements!
Where is the problem?
Added according to @JohnRennie 's answer:

The gravitational potential is not a fundamental property. The fundamental property is the geometry, and given some choice of a coordinate system the gravitational potential emerges from the geodesic motion.


Likewise the time dilation is a consequence of the geometry. So both the potential and the time dilation are a result of the geometry. It is not the case that the potential causes the time dilation.

I think these statements are somehow self-referential. We can also claim that it is the G-potential that determines the geometry around a G-mass as well as vice versa, which especially can be justified considering the fact that the coefficients in the Schwarzschild metric can easily be reformulated as functions of the G-potential.

Note that what matters here is the difference in the gravitational potential energy $\Delta\phi$ i.e. we set $\phi=0$ at the origin of our coordinate system then take the difference in the GPE relative to this point. The absolute value of the potential is not a physical observable.

You are right. However, the mentioned difference can be interpreted as the work done on a unit mass (the clock) to move it from infinity to the surface of the planet. I just cannot perceive how this work plays a decisive role in the clock rate.
For better understanding my issue, assume that we have a massive spherical shell. The G-acceleration is zero inside the shell as well as in infinity. The Schwarzschild observer located at infinity measures the rate of the clock located on the surface of the shell smaller than the same clock in his own hand. However, the observer at the center of the shell with similar feelings (zero G-field) to those experienced by the Schwarzschild observer, detects no change in the clock rate located on the shell compared to his because the potential difference is zero. This is slightly strange to me!
On the other hand, if there is an authenticity with the work done on the clock in GR, why general relativity predicts no change for the clock rates in E-fields (E-potentials) for charged clocks? That is, if we consider a massless shell though highly electrically charged and if use a charged clock, we may have to do the same work as we did on the chargeless clock in the previous example. However, this work can not affect the time rates for the clock located on the charged shell from the viewpoint of the Schwarzschild observer. Why this is the case?
 A: The gravitational potential is not a fundamental property. The fundamental property is the geometry, and given some choice of a coordinate system the gravitational potential emerges from the geodesic motion. That is, for any particular choice of coordinates the geodesic equation gives a coordinate acceleration that can be integrated to give a gravitational potential. Note that different choices for the coordinate system will give different gravitational potentials.
Likewise the time dilation is a consequence of the geometry. So both the potential and the time dilation are a result of the geometry. It is not the case that the potential causes the time dilation.
But as you say the time dilation is certainly correlated with the difference in gravitational potential energy. This happens because in the weak field limit the metric becomes:
$$ \mathrm ds^2 \approx -\left( 1 + \frac{2\Delta\phi}{c^2}\right) c^2~\mathrm dt^2 + \frac{1}{1 + 2\Delta\phi/c^2}\left(\mathrm dx^2 +\mathrm dy^2 + \mathrm dz^2\right) \tag{1} $$
You'll find derivations of this on the Internet, or there is a nice derivation for the specific case of the Schwarzschild metric in Derivation for the weak field Newtonian metric around Earth.
We get the time dilation by considering a stationary observer, i.e. $dx = dy = dz = 0$ in which case equation (1) gives relationship between the proper time for the observer and our coordinate time:
$$ \mathrm d\tau^2 = \left( 1 + \frac{2\Delta\phi}{c^2}\right)dt^2 \tag{2} $$
giving the time dilation:
$$ \frac{d\tau}{dt} = \sqrt{1 + \frac{2\Delta\phi}{c^2}} \tag{3} $$
Note that what matters here is the difference in the gravitational potential energy $\Delta\phi$ i.e. we set $\phi=0$ at the origin of our coordinate system then take the difference in the GPE relative to this point. The absolute value of the potential is not a physical observable.
