This answer is kind of parallel to Brandon's, because I want to emphasise the point underlying these types of observations.
We will never be able to observe a black hole, because for external observers the formation of an event horizon takes an infinite time. This may seem a bit pedantic, but it's an important point because our aim is not to directly observe a black hole but rather to measure the properties of a system and from these infer that the system must form a black hole.
For example, take Sagittarius A$^*$, which is believed to be a supermassive black hole at the centre of our galaxy. We can observe stars orbiting it and from these observations calculate that it's mass is about 4.1 million Solar masses, and its size is less than about 6.25 light-hours. This does not prove that Sag A$^*$ is a black hole, because the event horizon radius for the mass of Sag A$^*$ is about 40 light-seconds. However we don't know of any way an agglomeration of mass with this density could stay stable for anything like the lifetime of the Milky Way so we infer it must have formed a black hole. We expect future radio telescope measurements to make the limits on the radius even tighter and increase our confident that Sag A$^*$ must be a black hole.
Alternatively take Cygnus X-1. We can estimate it's mass using various methods and get a mass in the range 10-20 Solar masses. We can place a limit on it's size by measuring the timescale of changes in it's X-Ray emission, and we get an upper limit of about 10$^5$km (a bit less than the Sun). This makes Cygnus X-1 at least a neutron star, but assuming our calculations of stellar equations of state are reliable no neutron star heavier than 3 Solar masses can resist collapsing into a black hole. So once again we can infer the existance of a black hole even though we can't directly observe it.
Brandon's answer gives lovely examples of the measurements we can make to get evidence that black holes exist. The key point to understand with all of these is that we are trying to place lower limits on the density of the observed object. If these limits are sufficiently high, and assuming our current understanding of the physics involved is correct, then we can infer that the object is in the process of forming a black hole.