If you take apart a syringe you are left with one piece that is a tube with a tiny hole in one end and the open end of a tube at the other. If you then submerge this underwater and cover the small hole, then lift out of the water, the water remains in the tube/syringe until you release your finger from the small hole. Why is that?
This is because, when you close the end with the tiny hole in it, you prevent air from exerting any pressure on the water column. Thus the air pressure from the open end on the other side prevents it from falling. Once you take your hand away, you let air get into the syringe as a result of which the water falls.
An interesting thing happens when you close the tiny hole as the water is flowing down. The water doesn't stop instantaneously after you close the hole. It flows down a little more and stops. This is because even after you close the hole, the air pressure above the water is same as the air pressure below it. But soon, because of the flowing down water, the air column above expands resulting in a pressure drop stopping the water.