Your thoughts are basically right; the essential point is that sound waves travel through a medium at a certain speed, $c_s$, and as a result, there is an asymmetry between the effects due to the velocity $v_o$ of the observer relative to the medium and the velocity $v_s$ of the source relative to the medium, but no such asymmetry exists in the Doppler effect for light.
To get a handle on this, recall that quantitatively, the Doppler effect for sound is expressed by the following formula which relates the observed frequency $f_o$ to the frequency $f_s$ that one observes then one is at rest relative to the source;
\begin{align}
f_o = \frac{c_s+v_o}{c_s+v_s} f_s
\end{align}
The sign convention here is that $v_o$ is positive if the observer is moving towards the source, and negative if she is moving away, and $v_s$ is positive if the source is moving away from the observer, and negative if it is moving towards the receiver. To understand the asymmetry between observer and source velocity, suppose that $v_s = 0$, namely the source is standing still in the medium, then the relationship becomes
\begin{align}
f_o = \frac{c_s+v_o}{c_s} f_s
\end{align}
Now, notice that observer can make $v_o$ as high as one pleases (at least if we ignore the speed of light constraint) by moving through the medium towards the source as fast of one pleases. This allows one to observe arbitrarily high frequencies by moving faster and faster through the medium towards the source. On the other hand, if the observer speed is zero, then we have
\begin{align}
f_o = \frac{c_s}{c_s+v_s} f_s
\end{align}
and this time, attaining arbitrarily high observed frequencies involves moving through the medium with negative $v_s$ that has a magnitude close to $c_s$. In other words, on would have to arrange for the source to move towards the observer at a speed just below but very close to the speed of sound in the medium. This is completely different from what one needed to do when it was the observer moving towards a stationary source!
For light moving in vacuum in relativity, one does not encounter this asymmetry because light does not need to move through a medium, and the only thing that matters in the Doppler effect for light is the relative velocity between the observer and the source. In particular, there is no way to distinguish between the observer moving towards the source at a certain speed or the source moving towards the observer at the same speed.