It has nothing to do with action at a distance the way you talk about it.
There is some sort of instantanious action at a distance to the particles knowing where are holes to go ?
Consider that case of the barrier with vacuum inside. From one side, the air atoms are constantly colliding with the surface, and exerting a force on the barrier inward. From the other side, there are no atoms(or very few atoms, in case of low pressure) colliding with the barrier. So no force is exerted on the barrier outward.
This leads to a net inward force on the barrier due to air pressure, which in some cases leads to the barrier collapsing inward(unless it is strong enough to exert a force of itself to counter this inward force).
A similar thing happens if you have a hole in this barrier. An air atom moving towards the hole goes right in, because there is nothing on the other side of the hole stopping it. If there was air inside the barrier, the air atom moving towards the hole, would then collide with some air from the other side, and reflect back from the hole.
Although some atoms will go through, and even come out of the hole, the net effect will look like there is no exchange of air through the hole.
In a practical way, how the water in the ocean know that there is air inside the iron ship to pull it upwards out of water ?
The buoyancy of the ship has nothing to do with the pressure difference between the air and water, it is in fact due to difference in apparent density of the ship.