What is an Open Quantum System? The simple quantum text book examples like Simple Harmonic Oscillator potential and H-atom, seem to me open quantum systems, since the particle interacts with the potential. How is exactly the problem of open quantum systems different from these ones in which particles interact with a potential?
The examples you give are of two particle systems. The open quantum system presupposes a many body state.
The underlying nature of reality is quantum mechanical , thus all particles in the universe, in theory, should belong to one universal wavefunction solution, this would be the equivalent closed solution to the two body problems of your example. As this is an impossible task, models have been developed where there is a dominant solution for part of the matter under study, and the rest of the universe, the environment, is supposed to interact in higher orders, so as to be treated as a "bath", quantum mechanically. I.e. the matter under study is not "closed" quantum mechanically but is partly interacting with another quantum mechanical system.
The environment we wish to model as part of our open quantum system is typically very large, making exact solutions impossible to calculate. However, by making key approximations it is possible to find out how the quantum system behaves in the presence of the environment. There are two approximations commonly made in the field of open quantum systems: the Born approximation and the Markov approximation. The Born approximation assumes that the system-bath coupling is relatively weak, and the bath is very large, so that the bath is negligibly affected by the system. The Markov approximation assumes that the bath has no memory of past event
The distinction between an open and a closed quantum system is mostly about whether information about the system is copied into the outside world, not about interaction per se.
Suppose, for example, that a photon is reflected from a mirror. That is an interaction that changes the photon's momentum, but the interaction is not a measurement because the probability that it will produce a measurable change in the momentum of the reflecting object is extremely small. The reflector is in a mixed state in which its momentum has a range of values that is large compared to the momentum of the photon. So the shift in the mirror's momentum as a result of the reflection will be so small that it will not be detectable with very high probability. I don't know the exact numbers but let's say it's -1,000,000 to +1,000,000 in units of the momentum the photon will impart upon reflection. The photon is incident on the mirror and changes its momentum by +1 so that the range is now -999,999 to +1,000,001. So the probability is large that if you measure its state you won't detect any difference.
In the cases you describe above, for the discussion to be realistic, the potential has to be provided by a system that has a low probability of undergoing a measurable change. In a hydrogen atom, the proton is much larger than the electron and to some approximation it is not changed by the electron. The approximation is not perfect and the ways in which it fails have to be taken into account for many calculations, but a simpler model can explain some features of hydrogen atoms. Likewise if you want a system that provides a good approximation of the harmonic oscillator, whatever provides the potential should not be strongly affected by the system for which it provides the potential.