Wikipedia explains this rather well WIKI but I'll pick out the relevant stuff for you.
Before the Geiger–Marsden experiment the general idea was that atoms were build of some permeable positive substrate in which some negative particles were floating around, the so called plum pudding model.
If we shoot $\alpha$ particles on this setup they should all pass trough the atoms since the positive substrate is thought to be permeable ! (left side of the figure)
But when people did the experiment they saw that most particles went trough while some scattered $180^\circ$ backwards, some even bent a small angle ! (right side of the figure)
The plum pudding model had no problem with the particles that went trough undisturbed but what about the ones that were backscattered ? people theorized that there must be some solid core in the atom against which the $\alpha$ particles could scatter. The core couldn't be to big since only a small fraction of the $\alpha$ particles backscattered.
This leads to a model in which most of the mass (to which $\alpha$ particles can scatter) are in the center of the atom with the negative charges around it!