So if Rutherford predicted there is a high mass nucleus in the centre of the atom, if the particles hit the sides and rebounded up to 180 degrees, how come he did not measure whether particles hit the top of the nucleus and bounded upwards? If they did hit the top/bottom of the nucleus, could that mean that more atoms were being deflected (into the vertical plane) than they thought - thus his model not be as accurate as they found?
Here's a 3D model of the apparatus used by Rutherford, Geiger, and Marsden in their experiment.
In the middle is the gold foil; the light grey block to the immediate right is the chamber housing the radioactive source; and the tube to the left is the microscope for observing the scattered particles. The microscope can be rotated around the cylindrical chamber to observe scattered particles in multiple directions.
It is true that the microscope only rotated about the horizontal plane. But, imagine that the microscope could rotate vertically as well--perhaps to look down on the foil from above. This could also be achieved by just rotating the entire apparatus about the line joining the radioactive source and the center of the gold foil. Equivalently, imagine just rotating the gold foil about the same line. Nothing about the experiment changes. The particles are scattered in all directions, horizontally and vertically.
There is a symmetry to this experiment about the line about which the radiation particle are emitted (from the source to the center of the gold foil). Rutherford knew this and took this into account for his calculations. The angle in his scattering equation was the angle of deflection in any direction, horizontally and vertically. The same about of radiation would be deflected 45 degrees to the right as 45 degrees up. Because of this symmetry, the device used by Geiger and Marsden only needed to rotate in one plane (horizontally) to get an accurate sample of the scattered particles.