There are several points of evidence that the Oort Cloud exists, though it is indeed still a hypothesis and lacks direct observation.
The first is indirectly observational, as proposed by Ernst Öpik back in 1932 as the source of long-period comets. This was revised by Jan Oort in 1950. All you need to determine an orbit is three observations of the object, separated in time. The greater the separation in time and the more observations, the more certainty we have in its orbit. Comets with periods longer than Pluto's must, by definition, have come from beyond Pluto. Pluto's orbit basically loosely defines the extent of the Kuiper Belt (30-50 AU).
So there needs to be a source for these bound objects, and interstellar ones don't cut it because if they're interstellar, then they should not be on bound orbits.
The second is theoretical: Solar system formation models predict that the formation of the giant planets would have scattered small icy objects into the outer solar system. While some would be given enough energy to completely escape the solar system, others would be scattered out to the hypothetical Oort Cloud.
Third, we've seen Kuiper Belts around other star systems, and it's likely that the Oort Cloud is a continuation of the Kuiper Belt, so this may be evidence for Oort Clouds as well.
So if we need a source for long-period comets and the orbits work out to this cloud beyond the Kuiper Belt, dynamical models predict that the bodies would exist there, and we see similar dynamical structures around other stars, then that's fairly compelling evidence it exists.
But, you are correct that, at present, it is not technologically possible to view comets that are members of the Oort Cloud that are still in the Oort Cloud. Viewing a chunk of ice 1/4 of the way to the nearest star is simply not possible ... yet.