What is an open quantum system? What is an open quantum system? The simple quantum textbook examples like Simple Harmonic Oscillator potential and  H-atom, seem to me open quantum systems, since the particle interacts with the potential. How is exactly the problem of open quantum systems different from these ones in which  particles interact with a potential?
 A: The examples you give are of two particle systems. The open quantum system presupposes a many body state.
The underlying nature of reality is quantum mechanical , thus all particles in the universe, in theory, should belong to one universal wavefunction solution, this would be the equivalent  closed solution  to the two body problems of your example. As this is an impossible task, models have been developed where there is a dominant solution for part of the matter under study, and the rest of the universe, the environment, is supposed to interact in higher orders,  so as to be treated as a "bath", quantum mechanically. I.e. the matter under study is not "closed" quantum mechanically but is partly  interacting with another quantum mechanical system.

The environment we wish to model as part of our open quantum system is typically very large, making exact solutions impossible to calculate. However, by making key approximations it is possible to find out how the quantum system behaves in the presence of the environment. There are two approximations commonly made in the field of open quantum systems: the Born approximation and the Markov approximation. The Born approximation assumes that the system-bath coupling is relatively weak, and the bath is very large, so that the bath is negligibly affected by the system. The Markov approximation assumes that the bath has no memory of past event

A: The distinction between an open and a closed quantum system is mostly about whether information about the system is copied into the outside world, not about interaction per se.
Suppose, for example, that a photon is reflected from a mirror. That is an interaction that changes the photon's momentum, but the interaction is not a measurement because the probability that it will produce a measurable change in the momentum of the reflecting object is extremely small. The reflector is in a mixed state in which its momentum has a range of values that is large compared to the momentum of the photon. So the shift in the mirror's momentum as a result of the reflection will be so small that it will not be detectable with very high probability. I don't know the exact numbers but let's say it's -1,000,000 to +1,000,000 in units of the momentum the photon will impart upon reflection. The photon is incident on the mirror and changes its momentum by +1 so that the range is now -999,999 to +1,000,001. So the probability is large that if you measure its state you won't detect any difference.
In the cases you describe above, for the discussion to be realistic, the potential has to be provided by a system that has a low probability of undergoing a measurable change. In a hydrogen atom, the proton is much larger than the electron and to some approximation it is not changed by the electron. The approximation is not perfect and the ways in which it fails have to be taken into account for many calculations, but a simpler model can explain some features of hydrogen atoms. Likewise if you want a system that provides a good approximation of the harmonic oscillator, whatever provides the potential should not be strongly affected by the system for which it provides the potential.
A: Like many terms used in physics, open system may have somewhat different meaning depending on the particular field of research.
System in a thermostat
The definition described above is essentially system interacting with its surroundings, which in thermodynamic terms would be a system in contact with a thermostat. It is not quite clear why one came up with a special term for already known thing, but it is quite commonly used and this is how it will be understood in most cases. Perhaps, the difference with usual thermodynamic treatment is that the system may contain very few degrees of freedom - like a hydrogen atom or an electron. The term open here opposed to studying the same system as if it were isolated/closed from the rest of the world - e.g., as we solve the hydrogen atom in basic quantum mechanics.
Open, closed and isolated systems
The term open systems is often used when discussing non-equilibrium phenomena. In this sense, it is worth pointing out that textbooks on non-equilibrium thermodynamics, such as Modern Thermodynamics: From Heat Engines to Dissipative Structures by Prigogine, would often use more refined classification, which is at odds with the use described above:

*

*Isolated systems do not exchange energy or matter with exterior

*Closed systems  exchange energy with exterior but not matter

*Open systems exchange both energy and matter with exterior

Thus, the hydrogen atom described above in many cases should be considered a closed system, unless it emits electrons or, if we regard emitting and absorbing photons as a matter exchange.
A: Since the other answers consider a conceptual/qualitative description of open quantum systems, it might be helpful to add a formal description in addition.
In introductory quantum mechanics lectures, one typically considers a pure state $|\phi \rangle$ and a time evolution operator $U(t,0)$ such that $|\phi(t)\rangle = U(t,0) | \phi \rangle$. Here, typically $U(t,0) = e^{-i Ht}$ with $H$ the Hamiltonian.
However, such a description assumes that the quantum system is in isolation. In many experimental situations, the quantum system of interest is coupled to an environment, often times called bath in quantum thermodynamics. Let us denote our main system of interest by $S$. Since $S$ is coupled to the outside, we in general have to use density operators $\rho_S$ to describe the main system. The time evolution of $S$ is described by a quantum channel $\rho_S(t) = \mathcal C_t(\rho_S)$, i.e. a linear completely-positive and trace-preserving map.
Also, let us assume here we do not know the environment, as is the case in many experiments that consider this to be noise. Via procedures called purification and Stinespring's dilation theorem, it is always possible to introduce an abstract environment $E$ such that there are states $|\phi \rangle_{SE}$ with $\mathrm{Tr}_E |\phi \rangle \langle \phi |_{SE} = \rho_S$, and such that for a channel $\mathcal C$ with $\rho'_S = \mathcal C(\rho_S)$ there is a unitary $U_{SE}$ such that $\rho'_S = \mathrm{Tr}_E [U_{SE} \cdot |\phi\rangle\langle \phi |_{SE}]$.
In more physical terms, one can consider the system $S$ and its environment $E$ to be an isolated system. Then, the combined pure state $| \phi \rangle_{SE}$ evolves via a unitary $U_{SE}(t,0)$, and we get the system $S$ density operator $\rho_S (t)$ via the partial trace $\mathrm{Tr}_E$ over the environment.
The quantum channel $\mathcal C_t$ describing the evolution of $\rho(t)_S$ is typically assumed to be described by a so-called Master equation in Lindblad form:
$$
\frac{\mathrm d}{\mathrm d t} \rho_S = -i [H_S, \rho_S] + \sum_a L_a \rho_S L^\dagger_a - \frac{1}{2} L^\dagger_a L_a \rho_S - \frac{1}{2} \rho_S L_a^\dagger L_a
$$
Here, the term $-i [H_S, \rho_S]$ is sometimes called the coherent part, while the terms involving the $L_a$ are typically used to model noise and decoherence.
To answer your question about the hydrogen atom and the harmonic oscilator: Here, the potentials are effective/reduced descriptions like the $H_S$ in the Master equation. This means the origin/source of this potential is not explained. For the hydrogen atom, a proper description in quantum electrodynamics would treat the electromagnetic field as a quantum field. And then this quantum field would be an environment for the atom. Usually, such a refined description can be coarse-grained to just a Coulomb potential. However, the quantum field is necessary to understand the fine-structure of the hydrogen atom. Similarly, the harmonic oscillator usually has its origin in quantum field theory. Here, it arises as an approximation/effective description that applies if one can neglect the interaction between the fields and the modes (i.e. in the free field limit).
