Think about pressure in a cylinder containing gas at high pressure. If the cylinder is closed then the pressure is acting everywhere on all the walls of the cylinder. Now open one end of the cylinder. The gas at that end rushes out (and does not push on the cylinder). The gas at the other end is still pushing on the cylinder with a force $pA$ where $p$ is the pressure in the gas, and $A$ is the area of the end wall. As the gas escapes $p$ falls, but it can be quite high to begin with.
Similar things go on inside a rocket motor. The main difference is that fuel is fed in and burned so as to maintain the pressure.
Comment on Newton's third law
I would say it is a little unclear to appeal to Newton's third law in the case of a rocket. It is better to invoke conservation of momentum, but these two ideas are closely related. The third law is acting here at the collisions between gas molecules and wall of cylinder, and at the collisions among the gas molecules. The conservation of momentum is also acting in all those processes, and it can be invoked for the overall situation: the backwards momentum of the escaping gas is exactly balanced by the forwards momentum of the rocket.
Similar remarks apply to the recoil of a gun. The bullet does not cause the recoil. Rather, the explosion of the gunpower produces a large force on both the bullet and the gun. It is this force that causes both the acceleration of the bullet and the recoil of the gun. However, from the fact that a bullet is speeding away you can deduce that the gun must be recoiling.
The reasoning is like detective work. Sometimes you will see puddles of water forming on the ground, and at the same time people are putting on their coats and opening their umbrellas. Did the coats and umbrellas cause the puddles then? No. Rather, it has started raining and this caused both the puddles and the umbrellas. Similarly, the bullet does not cause the recoil. And for the rocket it is not the exhaust gas but the high-pressure gas inside the rocket which is causing the rocket to accelerate at any given time, by exerting a force on it.