I know this is a very basic (and probably stupid) question, but I have been confused about this for a long time.
Both Ignacio and Anna's answers are correct, but let me see if I can expand on them a little.
Why does a gas inside a closed container experience atmosperic pressure when the gas itself is not in contact with the atmosphere?
but actually a gas inside a closed container doesn't necessarily experience atmospheric pressure. If you've ever shaken a bottle of pop then opened it you'll know that the gas pressure inside is a lot higher than atmospheric pressure. This is an example where the pressure inside the closed container is higher than atmospheric pressure, but it can be lower as well.
What happens to the gas in a container depends on how flexible the container is. If I take a glass bottle and pump half the air out, the gas inside will stay at half atmospheric pressure. That's because the glass walls of the bottle are rigid and they take some of the pressure.
On the other hand if I fill a plastic bag with gas then pump some out, the pressure inside the bag stays at atmospheric pressure. That's because the plastic bag is flexible. The atmosphere presses on the bag and the bag transmits the pressure to the gas inside it.
How do you define "contact"?
For gases contact does not mean "touching".
Small leaks, as small as molecular sizes of the air molecules, will allow leaking and equalization of pressure with the atmospheric pressure.
It needs good seals to be able to keep a vacuum, i.e. a container where no air exists, but it can be done. If the seals are not good, then pressure equalization takes place.
Imagine the gas is sealed inside a piston, like this:
The piston head is free to move in the sleeve, which changes the volume of the gas. We know from experience that the piston head will rapidly find an equilibrium position where it doesn't move any more. If the piston has stopped moving then that means it's not accelerating, and if it's not accelerating that means that the forces acting on it must sum to zero.
Both the gas inside the piston and the atmosphere outside it exert a force on the piston. The magnitude of these forces is equal to the pressure of the gas (or of the atmosphere), times the area upon which it acts. The two forces act in opposite directions, and the piston has the same area on each side, which means that the two pressures have to be equal. This is the reason that gases inside containers reach the same pressure as the air outside - the volume of the gas will adjust until the forces acting on the container are balanced.
Of course, if there other forces than pressure acting on the container wall then the pressures will not be equal. This is why it's possible to create a (partial) vacuum inside a rigid container. It's also why the air inside an inflated balloon has a higher pressure than the air outside it - the balloon's skin is subject to a tension force as well as the two pressure forces.
You should realize that momentum is a physical thing--- it is a conserved quantity like energy. Momentum flows from place to place, like electric charge, but because it is a vector quantity, the flow is harder to imagine. If you look only at the x-component of momentum, this flow is easier to visualize.
So, for example, if you push on a block, the momentum is flowing through your hand to the block and builds up on the block, making it go faster. If there is friction, the momentum leaks back through the ground to the earth, and the friction between your shoes and the ground completes the x-momentum circuit. When you have a mass sitting on a table, the y-momentum comes from the Earth to the mass, and flows down through the table to the Earth.
Pressure is when the x-momentum is flowing in the (positive) x-direction only, the y-momentum is flowing in the (positive) y-direction only, and likewise for the z-momentum. This is the intuitive idea, since if you put two y-z plates at x=0 and x=a, you need to absorb as much x-momentum as hits the plate on the left and leaves the plate on the right, which means you need to hold the plates apart with a force proportional to their area, and the proportionality constant is equal to the momentum flow rate.
When you rotate the system, you rotate the x,y,z components, but you also rotate the direction of the flow, and the two rotations together give you a tensor transformation law. The pressure is the only rotationally invariant mometum flow possible, and in fluids, rotational invariance is still a correct symmetry, so they can only have a pressure when they are in equilibrium.
When you have gas in a container, the three flows of momentum are passing right through the walls of the container through the gas and out the other side. So the gas has the same momentum-current inside as out, the same pressure inside as out, since the momentum is passing right through the walls. If you want the gas to be at lower pressure, you need to deflect the momentum currents around the gas on the interior, so you need to make a stiff box that pushes the momentum current around the interior. In this case, the box has to have large shear stresses (deflections of x-momentum in the y and z directions) to get the momentum to the other side without going through the middle.
This point of view is the modern stress-tensor view of Newton's laws, and it makes the answer to this question completely obvious, but it isn't emphasized in elementary books, probably because tensors are considered advanced. A tensor is just a vector of vectors, a vector whose x component, y component, and z component is a vector, and there is no reason to delay learning about it.