Working out how much force is required to move through a fluid, whether it's air or water, is surprisingly difficult because there are two effects you need to take into account. However neither of those effects is pressure (or at least only indirectly related to pressure).
The first effect is viscosity, which is simply how thick the fluid is. Obviously it's harder to move through a viscous fluid like treacle than through a thin fluid like water. Non-zero viscosity means that as you move you feel a viscous force.
The second effect is inertia. If you move your hand through water then the water in front of your hand has to start moving so it can flow out of the way. This means you have to accelerate the water, and because water has a mass Newton's first law tells us you need to apply a force to accelerate it. You feel this force as an inertial force.
However neither of these forces directly involve pressure. There is an indirect effect for compressible fluids like air because as you increase the pressure the air becomes more dense. As the air density increases its viscosity increases so you feel an increased viscous force, and its density also increases so you feel an increased inertial force. So all that pressure does indeed make the air harder to move through, but only indirectly.
If you now consider water, this is almost incompressible and its density doesn't change much over the pressure range humans can tolerate. This means the force required to move through water changes hardly at all with pressure. You say it's harder to swim the deeper you go, but then scuba divers can swim down to 300m depth where the pressure is about 30 times the pressure at the surface - admittedly 30m is a more sensible limit for recreational diving, but that's still 3 times the pressure at the surface.
