Pressure is synonymous with energy - its a kind of kinetic potential energy, not unlike heat. Kinetic theory relates these two quantities, together with volume.
Let's make some assumptions. When we bottle something we expect its volume to stay about the same - the glass shouldn't flex. Also we pretend the bottom of the ocean is a comfy room temperature, so that heat is a constant too.
Now it might make more sense why pressurisedness (energy) doesn't change. Where could that energy go?
The confusion lies in what incompressible means. Compressibility relates changes to volume with changes in pressure/potential energy.
Consider a piston containing water and one containing air. Recall that energy = force * distance.
If we pressurised the air, it would compress easily at first. The force would be small and the distance large. Eventually the piston would reach our desired pressure, and the force would reach a maximum (when the air pressure balances it).
If we pressurised the water, it's volume would barely change. The force would be large and applied over a very short distance. However the same pressure (and force on the piston) would be reached.
This means that both pistons contain the same potential energy! They are just as equally compressed despite very different volumes. How come we don't see more pressurised solids in day to day life? I think this might be because they just can't compressed as easily as water or liquid. The most common application is the spring, where we call this pressure "stress". Engineers well now that not many solids behave very "elastically" - when they are suitably compressed (compression is called "strain" here), they are likely to deform/bend (energy loss) rather than return to their original shape.
See the similar question about why iron balls don't bounce.