Adding matter to make galaxies bigger What are the actual factors that play a role in the accretion of matter into galaxies? I read about Accretion Disks but I don't quite understand how they work yet. 
 A: I will select quotes from the wiki article on structure formation, bold mine:

The very early Universe
In this stage, some mechanism, such as cosmic inflation, is responsible for establishing the initial conditions of the Universe: homogeneity, isotropy and flatness.3[6] Cosmic inflation also would have amplified minute quantum fluctuations (pre-inflation) into slight overdensities (post-inflation). These acted as seeds around which the dark matter could begin to gravitationally congregate, even as the normal, baryonic matter was still in thermal equilibrium — far too hot to allow gravity to get any purchase on it.

The basic component of galaxy formation is the force of gravity, as we know it in its newtonian form. But if the universe had remained homogenous there would not have been preferred locations in the expanding space that would feel the gravitational attraction more than other locations, so that aggregates of denser matter could evolve. It is the **quantum fluctuations **    that generate spots of inhomogeneity where gravity forces would be stronger locally in that space than in contingent ones. 

Finally, at about 400,000 years after the 'bang', it's cool enough for the nuclei to capture their electrons, forming neutral-charge atoms. As the charged particles pair up, the photons no longer interact with them, they are free to propagate, and we currently detect those photons as the Cosmic Microwave Background Radiation (CMB), because they fill the Universe.

The CMB is a snapshot of that time and it does show the inhomogeneities hypothesized by the proposal of quantum fluctuations. 

In those clusters nuclei have already formed and the higher gravitational fields in the clusters will slowly coalesce into galaxies and clusters of galaxies by the gravitational force, which though very weak is inexorable and acts at large distances.
This outline is the most accepted  model of how the structure of galaxies and clusters of galaxies has formed though the theoretical calculations which have to include General Relativity can still be open to research.  After all we still do not have a consistent unified model of a quantized gravity in  a unified manner with the other three forces.
That is how galaxies are seeded, by the primordial quantum mechanical fluctuations and the consequent coagulation of matter within them, in time.
Now accretion is a formation proposal  used for single stars and planets, within galaxies.

n astrophysics, accretion is the growth of a massive object by gravitationally attracting more matter, typically gaseous matter in an accretion disk.1 Accretion disks are common around smaller stars or stellar remnants in a close binary or black holes in the centers of spiral galaxies. Some dynamics in the disk are necessary to allow orbiting gas to lose angular momentum and fall onto the central massive object. Occasionally, this can result in stellar surface fusion

So you are applying a wrong model to galaxy formation.
