The pressure is impulse the air molecules give you in collision on unit area and therefore depends on their speed and density. For simplicity, i will consider only density (this would mean that air column has everywhere the same temperature. This is not true for our atmosphere, but such scenario could be achieved in a lab).
Because of gravitation, the molecules of air are trying to get as close to the ground as possible - and the more of them there are, the more collisions with you will happen and the bigger the pressure.
However, the air molecules collide with each other and in the collision the direction of molecules movement changes - and the more crowded the ground is, the more molecules will get repelled from the ground to the higher layer, which is less crowded. But it is crowded, so some molecules from the higher layer will fall down to the ground layer also.
Now there is some balance in density of molecules between these layers, where the amount of molecules that goes into some layer is the same as those that get repelled from the layer. Without gravitation this simply means, that the density should be everywhere the same. In that case all layers will have the same amount of molecules repelled from their layer, but since this amount is everywhere the same, so is the amount of accepted molecules same. But with gravitation, the molecules are getting some help falling from higher layer to lower layer, so you need smaller density in higher layer in order to have the same amount of molecules repelled from lower layer upward and fallen from higher layer downward.
This balance depends on the whole profile of the air column. If you would have only two layers - the ground (first) and the above (second), then soon the third layer above the second would be created from the molecules repelled from second layer upward. Since there is no third layer at the beginning, the molecules would not fall from the third layer to the second layer and the second layer would be loosing molecules/its density. Since it looses density, the ground layer will have less and less molecules returning from the second layer and the ground layer would also loose density and therefore also the pressure. This will go on until the balance is created, and this balance depends on all of the layers, not just the one you are considering.
The individual molecules that are colliding with you have no idea what happens above them. But the density of molecules on the ground, depends on how dense are the layers above - considering the balance was created.
Also, you misinterpreted the formula for pressure.
Imagine column of molecules of same type with mass $m$, all going right with the same velocity $v$, where they hit wall and return back with same velocity. The pressure for this situation is this:
$$P=\frac{F}{S}=\frac{N\Delta p}{S\Delta t}=\frac{2Nmv}{S\Delta t}$$
where $S$ is the area of the wall that is being hit, the $\Delta p$ is momentum change of individual molecule, and N is amount of molecules that hit the area $S$ in time $\Delta t$. How many is this? Well, since we assume constant density, there is $N_V$ molecules in volume $V$ of the column. This column moved to the right with velocity $v$ so the volume of the column that "moved through the wall" in the time $\Delta t$ is $V=Sv\Delta t$. From this you get the formula:
$$P=\frac{2N_Vmv^2}{V}$$. But since the air is not column of moleculs moving to the right and molecules have random directions you need to average them (that is, divide by 6 directions). Then you also need to average velocities and you will get your formula:
$$P=\frac{N_Vm\bar{v}^2}{3V}$$
But how did you get the volume? You got it in the process of trying to eliminate counting the number of molecules hitting the wall using time interval and area of the wall. Instead you used the fact that density is constant and rewritten the equation in such a way, that eliminates the information about some random time interval and area of the wall. But, the formula in this form is not general, because the volume is used to count the number of molecules hitting the wall, not to count all the molecules in the gass. So the formula cannot be used in a way you are trying to use it. It can be used if you have gass in some jar, where there is constant density of the gass everywhere. Then it is useful, because you can directly manipulate the volume, but you cannot directly manipulate the density. And since amount of molecules is the same (unless the jar is opened) it is more useful having $N/V$ in the formula instead of number density, which would be more appropriate from theoretical standpoint.
