How can anything ever fall into a black hole as seen from an outside observer? The event horizon of a black hole is where gravity is such that not even light can escape. This is also the point I understand that according to Einstein time dilation will be infinite for a far-away-observer.
If this is the case how can anything ever fall into a black hole.  In my thought experiment I am in a spaceship with a powerful telescope that can detect light at a wide range of wavelengths.  I have it focused on the black hole and watch as a large rock approaches the event horizon.
Am I correct in saying that from my far-away-position the rock would freeze outside the event horizon and would never pass it? If this is the case how can a black hole ever consume any material, let alone grow to millions of solar masses.  If I was able to train the telescope onto the black hole for millions of years would I still see the rock at the edge of the event horizon?
I am getting ready for the response of the object would slowly fade.  Why would it slowly fade and if it would how long would this fading take?  If it is going to red shift at some point would the red shifting not slow down to a standstill? This question has been bugging me for years!
OK - just an edit based on responses so far.  Again, please keep thinking from an observers point of view.  If observers see objects slowly fade and slowly disappear as they approach the event horizon would that mean that over time the event horizon would be "lumpy" with objects invisible, but not passed through?  We should be able to detect the "lumpiness" should we not through?
 A: It is true that, from an outside perspective, nothing can ever pass the event horizon.  I will attempt to describe the situation as best I can, to the best of my knowledge.
First, let's imagine a classical black hole.  By "classical" I mean a black-hole solution to Einstein's equations, which we imagine not to emit Hawking radiation (for now).  Such an object would persist for ever.  Let's imagine throwing a clock into it.  We will stand a long way from the black hole and watch the clock fall in.
What we notice as the clock approaches the event horizon is that it slows down compared to our clock.  In fact its hands will asymptotically approach a certain time, which we might as well call 12:00.  The light from the clock will also slow down, becoming red-shifted quite rapidly towards the radio end of the spectrum.  Because of this red shift, and because we can only ever see photons emitted by the clock before it struck twelve, it will rapidly become very hard to detect.  Eventually it will get to the point where we'd have to wait billions of years in between photons.  Nevertheless, as you say, it is always possible in principle to detect the clock, because it never passes the event horizon. 
I had the opportunity to chat to a cosmologist about this subject a few months ago, and what he said was that this red-shifting towards undetectability happens very quickly. (I believe the "no hair theorem" provides the justification for this.)  He also said that the black-hole-with-an-essentially-undetectable-object-just-outside-its-event-horizon is a very good approximation to a black hole of a slightly larger mass.
(At this point I want to note in passing that any "real" black hole will emit Hawking radiation until it eventually evaporates away to nothing.  Since our clock will still not have passed the event horizon by the time this happens, it must eventually escape - although presumably the Hawking radiation interacts with it on the way out. Presumably, from the clock's perspective all those billions of years of radiation will appear in the split-second before 12:00, so it won't come out looking much like a clock any more. To my mind the resolution to the black hole information paradox lies along this line of reasoning and not in any specifics of string theory.  But of course that's just my opinion.)
Now, this idea seems a bit weird (to me and I think to you as well) because if nothing ever passes the event horizon, how can there ever be a black hole in the first place?  My friendly cosmologist's answer boiled down to this: the black hole itself is only ever an approximation.  When a bunch of matter collapses in on itself it very rapidly converges towards something that looks like a black-hole solution to Einstein's equations, to the point where to all intents and purposes you can treat it as if the matter is inside the event horizon rather than outside it.  But this is only ever an approximation because from our perspective none of the infalling matter can ever pass the event horizon.
A: I would like to add a fact that, perhaps, is not controversial.
Namely, that all the information about any infalling object will be available for the outside observer at any time. The information cannot get lost under the horizon, otherwise we have the information loss paradox.
This means that it is theoretically possible for an outside observer to restore any object that went in the direction of the BH, because all of its information still kept.
This is true not only regarding objects that are falling after BH formation but also for those objects which were there at the time the star collapsed. So even if you were in the center of a star when it was collapsing, all information about you is still preserved, available outside the horizon and your body can be reconstructed.
A: Seems to me the faller is part of the black hold and so will itself be evaporating
If one throws a log in a fire is it the fire that burns the log, or is the log now part of the very fire.
 I see the faller as part of the event horizon, so rather than say the faller is destroyed by a fire-wall, maybe the faller itself evaporates.
Perhaps this is just quibbling over semantics.
A: Assume the object falling in is a blue laser that you launched directly (radially) towards the Schwarzchild (non-rotating) black hole that is aimed directly at you and that you are far from the black hole. The massive object is the laser itself, the light that you are watching is your way to "see" the object as it approaches the event horizon.
First of all just because the laser is moving away from you it will be slightly red-shifted just by the Doppler effect.  As it approaches the black hole that slight red-shift will become more and more significant.  The laser light will go from blue, to green, to yellow, to red, to infrared, to microwave and to longer and longer wavelength radio waves as it appears to approach the event horizon from your point of view.  Also the number of photons it emits per second (as you detect them) will decrease with time as the horizon is approached.  This is the dimming effect - as the wavelength increases, the number of photons per second will decrease.  So you will have to wait longer and longer between times when you detect the longer and longer wavelength radio waves from the blue laser.  This will not go on forever - there will be a last photon that you ever detect.  To explain why, let's look at the observer falling in.  
Your friend who is the observer riding on the laser does not even see anything happen when he crosses the event horizon (if he is freely falling).  The point is that the event horizon is not at all like a surface that you hit or where anything unusual happens from the freely falling observers point of view.  The reason why there will be a last photon you will ever detect is because there are only a finite number of photons emitted between the time the laser starts to fall and the time the laser crosses the event horizon.  So that last photon emitted just before it goes over the event horizon will be the last photon you will ever see. That photon will be a very long wavelength photon and you may not see it until some time in the distant future - how far in the future will depend on the number of photons per second that the laser emits - but there will be a last photon and after that you will not see any more photons.
So, I claim that the laser does disappear from an outside observer's point of view.  Note that trying to "illuminate" the object near the event horizon by shining a different laser on the object and looking for scattered photons will not work. (It will not work even if you throw the second laser in to try to illuminate the first laser.) From the point of view of the laser that fell in, these photons will only hit the laser after it has already crossed the event horizon and therefore the scattered light cannot escape from the black hole.  (In fact, if you wait too long before you try to illuminate the object, the infalling laser will have already hit the singularity at the center of the black hole.) From the outside observer's "point of view" (but he cannot "see" this), the infalling laser and the photons that are trying to illuminate the laser will get "closer and closer" to each other as they get frozen on the event horizon - but they will never interact and there will never be a scattered photon that you might try to detect.
A: Everything you say in your question is true, and your comment "the event horizon is in a different time reference" is also true, though it needs to be stated more precisely.
If you've read much on relativity you've probably come across terms like "frame of reference" and "inertial frame". A "frame" is a coordinate system, i.e., a system of distances, angles and times used to measure the location of things. For example, the map grid references are a coordinate system used to measure locations of things on the Earth's surface.
General relativity (GR) gives us a way to describe the universe that is independent of any frame of reference. However, for us observers, to calculate what we see, we have to do the calculations in our frame of reference i.e. in meters and seconds that we can measure. The static black hole is described by the Schwarzschild metric, and it's not hard to use this to calculate things like how long it takes to fall onto the event horizon. One common coordinate system is co-moving coordinates, i.e., the observer falling into the black hole measures distances from himself (putting himself at the origin) and time on the stop watch he's carrying. If you do this calculation you find the observer falls through the event horizon in a finite time, and in fact hits the singularity at the centre of the black hole in a finite time.
But where things get odd is we calculate the time taken to reach the event horizon in our coordinate system as observers sitting outside the black hole. This is an easy calculation, that you'll find in any introductory book on GR, and the answer is that it takes an infinite time to reach the event horizon.
This isn't some accounting trick; it means we will never see an event horizon form. At this point someone will usually pop up and say that means black holes don't really exist. In a sense that's true in our coordinate system, but all that means is that our coordinate system does not provide a complete description of the universe. That's something we've been getting used to ever since Galileo pointed out that the Sun doesn't revolve around the Earth. In the coordinate system of the freely falling observer the event horizon does exist and can be reached in a finite time.
You ask:

If this is the case how can a black hole ever consume any material, let
alone grow to millions of solar masses.

As long as you stay outside the event horizon a black hole is nothing special. It's just an aggregation of matter like a star. In the centre of our galaxy we have a compact region, Sagittarius A*, containing millions of star masses, and from the orbits of stars near Sagittarius A* it contains enough matter in a small enough space to make it a black hole. However, the orbits of those stars just depend on the mass they're orbiting and whether Sagittarius A* is actually a black hole or not is irrelevant.
A: Indeed, nothing can get under the horizon. The stuff close to the event horizon does move outwards as the black hole radius increases. Even more with any black hole deformations such as waves on its surface, the tidal deformations or the change of the rotation speed, all the objects close enough to the horizon remain "sticked" to it and follow all the changes of the black hole form. All objects close enough to a rotating black hole horizon, rotate with it at the same speed. If a black hole moves, so does everything close enough to its surface, including the things located on the side of the direction of the move. If anyone interested what mechanism make such sticking possible, it is called frame-dragging.
You may ask then, how a black hole can appear then and the horizon form. It is conjectured that they cannot, and the only possible black holes are the hypothetical primordial black holes that existed from the very beginning of the universe.
The objects that can be very similar to black holes are called collapsars. They are virtually indistinguishable from actual black holes after a very short time of the formation. They consist only of matter outside the radius of the event horizon of a black hole with the same mass. This matter is virtually frozen on the surface like with actual black hole, due to high gravity level.
Such collapsars possibly can become black holes for a short time due to quantum fluctuations and thus emit hawking radiation.
Astrophysicists do not separate such collapsars from actual black holes and call all them black holes due to practical reasons because of their actual indistinguishability.
Here is a quote from one paper that supports such point of view:

Our primary result, that no event horizon forms in gravitational
collapse as seen by an asymptotic observer is suggestive of the
possibility of using the number of local event horizons to classify
and divide Hilbert space into superselection sectors, labeled by the
number of local event horizons. Our result suggests that no operator
could increase the number of event horizons, but the possibility
of reducing the number of pre-existing primordial event horizons is
not so clear and would require that Hawking radiation not cause any
primordial black hole event horizons to evaporate completely.

Source
A: There is a good answer by John Rennie, and I think the continued discussion in the comments rests on a misunderstanding by the OP, who asks in a comment to John:

"How can anything ever fall into a black hole as seen from an outside observer?" – Matt Luckham Feb 24 '12 at 10:19

The misunderstanding of the OP is in the definition of the "outside observer" and in assuming that all observers are "outside".
"Outside" means outside the gravitational influence/pull of the black hole, i.e., not attracted and falling into it. This of course does not define "all observers". There will be observers falling into the black hole because their trajectory falls towards the black hole, no matter how far they are. This includes all the matter falling towards the singularity, seen  from whatever framework.
A: I recommend reading the answers to some of the questions at right --> -->
Particularly this one.
I expect that this question will be closed as exact duplicate, but what you'll find in response to the other questions is that what someone falling into a black hole observes, and what someone outside who is watching them fall in, are not the same.  The exact nature of the change of the image can be (and has been) worked out, but again, I recommend looking at the other questions here.
A: Third paragraph onwards are slightly speculative on my part; I'm not exactly sure of them. Comments appreciated
This is all due to the queerness of relativity. In your reference frame, the rock stops at the horizon. The rock senses no such stopping. The rock will see the stars condense (this 'condensation' is more apparent for massive black holes), due to gravitational lensing. It will see the horizon approaching, and will fall through it. 
Remember, time and space are relative. This is a rather extreme case where time appears to flow infinitely faster in a different frame.
About the 'dimming', I'm not exactly sure what happens. IIRC, the claim that the rock 'freezes' is a half-truth. Theoretically, the rock is frozen in your reference frame, but a telescope can't see that. To make life easier, let's assume the rock to be covered in lamps (a normal rock would become invisible long before it reaches the horizon)., At the horizon, the light emitted by these lamps is $\infty$ redshifted, so it basically doesn by exist.  This can also be looked at as the photon turning tail and being absorbed (actually the photon gets frozen at the horizon) So a rock at the horizon is invisible. A rock near the horizon is very dim, as almost all the light emitted is re-absorbed(also, there is redshifting of light, more redshift $\implies$ less energy. ). So, what we see is that the rock gradually dims as it reaches the horizon. It also appears to go slower. The rate of dimming and slowing converge at the horizon, where the rock is frozen, but completely invisible. IMHO, this happens at $t=\infty$ in your frame.
So a black hole stays black. You won't see any stars, gas, rocks, or ambitious researchers stuck to the horizon, though you may manage to see dim versions of them as they fall in near the horizon.
As to how the black hole grows, its due to the absolute horizon. The horizon of a black hole grows in 'anticipation' of infalling material.
A: One version I heard is this : the radius of the event horizon can be defined using the mass it envelopes. Now although from an outside observer, an object never enters the event horizon, but in a finite time it will be very very close to it. Now if you include this object as part of the black hole and recalculate the event horizon, you'll find that the new event horizon already includes this object, therefore the object can be seen as inside the newly formed black hole.
A: This is most easily seen from the mathematical point of view:
the Schwarzschild metric reads:
$$ds^2=-c^2(1-\frac{2GM}{c^2r})dt^2+(1-\frac{2GM}{c^2r})^{-1}dr^2+r^2d\theta^2+r^2sin^2\theta d\phi^2$$
for bodies crossing the horizon slowly, we may have
$$\frac{dt}{d\tau}=\frac{1}{\sqrt g_{00}}=(1-\frac{2GM}{c^2r})^{-1}$$
The radius of a black hole is its Horizon, and this radius is the Schwarzschild Radius, which equals $r=\frac{2GM}{c^2}$. So at the horizon, the ratio of the coordinate time and the proper time is infinite, which means you would measure the bodies' time to slow down by a factor of infinite, which means you wll see them stop at the horizon.
A: At the event horizon, the person getting sucked in will see light twice as fast. Then as he falls in he will eventually see light 3 times as fast, then 4 times as fast, then 5 times as fast, until light seems to be infinitely fast and time starts to grow infinite to him, and probably he will seem to take a split-second to fall in, then he will be destroyed because there is only space for less than a cubic-planck (infinitesmall) while he's bigger than a cell. However outside, you'll never see something falling in the event horizon, and when the escape velocity needed is half the speed of light, you will see objects going half the speed, which isn't so bad, but after a while, when the speed needed becomes 299, 792, 457 meters per second, and pretend there's a clock, it will be 1/299, 792, 458 times as fast and therefore take around 9 years to go 1 second, which probably the clock already fell in. The person falling in, however feels nothing special when crossing the event horizon, and can even communicate with another person falling in, until one is destroyed by gravity singularity. There's no way to go into a black hole and tell the tale to everyone, but there's a way to identify every object that has fallen in the black hole since the big bang, but you'll need a super duper fast running program and much better eye-sight than normal. Since the faster you go the slower time is for you because the speed of light is only a little faster than you, you'll have to wait until you're almost the speed of light like 299, 792, 457.99[...]9 meters per second, then immediately without waiting 299, 792, 458/ the number I wrote -1  seconds, and move back at the speed of light all the way for a very, very long time, probably more than the time big bang took to form and tell the tale. And no, you age at the same speed, but time just SEEMS much slower because light is only going a bit faster than you.
A: You see objects freezing outside the event horizon. You see event horizon moving outwards when more stuff falls into the event horizon. Stuff in the event horizon does not move outwards, when event horizon moves outwards, therefore objects become engulfed by the event horizon. 
A: If my knowledge of time dilation is correct, it goes both ways. Time slows down for the object in this situation.  But, time does not slow down for an outside observer.  Therefore, no, time would be infinite only for the object nearing the black hole. (I might be wrong, if I am, please tell me in the comments)
A: The problem with testing if anything but the image/information remains floating above where the event horizon should be is that any signals an object emits get both slower and red-shifted to the point where a probe will become essentially unresponsive and invisible. One possible way to verify if objects are really still hovering there is if you equipped a bunch of test probes with mirrors that reflect short wavelengths really well (the reason to have a bunch of them is that eventually, even single photons will transfer a destructive amount of momentum). I'm guessing you will get reflections as long as you don't run out of probes (though you WILL have to wait exponentially longer for each additional measurement).
Edit: Based on a comment by Anixx, the probes won't be "swallowed" as their mass expands the black hole, but will simply be pushed outward slightly.
