You're right, it's basically because of angular momentum. In essence, if you start with a self-gravitating cloud of material or collection of particles with a mean angular momentum (which needn't be particularly large), then the material smears itself out perpendicular along the plane of rotation (perpendicular to the axis of rotation). The individual motions perpendicular to the plane roughly cancel out through assorted interactions. But, at least on average, the material is all orbiting in roughly the same direction, so that component is preserved. In even broader terms, the evolution allows energy to be lost (through collisions, heating, etc.) but losing angular momentum is much more difficult.
If it were the other way round (i.e. losing energy is difficult, angular momentum easy) then we might expect spherical clouds. For example, in dark matter halos, it's very difficult to lose energy because dark matter cannot radiate energy away, so they remain diffuse and more broadly distributed. i.e. they don't collapse into discs. Giant ellipticals are thought to be the remnants of mergers between massive galaxies and the random orientations of the input angular momenta mean that the remnant has a smaller angular momentum relative to its energy.
Beware of placing too much importance on the central object, though. In the case of, say, an accretion disc around a compact object (white dwarf, neutron star or black hole), the central object totally dominates the behaviour of the orbiting material. In the Solar System, the central object (the Sun) mostly dominates the orbital behaviour but clearly there are smaller systems where other objects rule, like planets over their moons. In the Milky Way, the central black hole actually only dominates over a small region in the centre. Our orbit is determined by the black hole and all the stars, gas and dark matter inside our orbit. It doesn't affect the description above but I thought it was worth saying.
It'd be really great if there was an animation of the "smearing out" of a sphericalish cloud into an accretion disc but I couldn't find one...