Edit: As requested, here is a calculation for the ellipsis. An ellipsis can be parametrically described by $\vec r(t) = (a\cos \varphi(t),\, b\sin\varphi(t))^\top$ where $0\leq\varphi(t) < 2\pi$. Then,
$$\vec v(t) = \dot{\vec{r}}(t) = (-a\dot{\varphi}(t)\sin\varphi(t),\, b\dot{\varphi}(t)\cos\varphi(t))^\top.$$
Now take the dot product
$$
\vec{r}(t)\cdot\vec{v}(t) = -a^2\dot\varphi(t)\sin\varphi(t)\cos\varphi(t) + b^2\dot\varphi(t)\sin\varphi(t)\cos\varphi(t).
$$
We can see, that it vanishes for all times $t$ if and only if $a=b$ (i.e. the orbit is circular).
The position and velocity are not always perpendicular to each other for an elliptical orbit. This can be easily seen if you draw an ellipsis and add the position and velocity vectors.
Actually, the position is perpendicular to the velocity if and only if the orbit is a circle. This is because only then the radius remains constant, which is in fact the definition of a circle. If it is not the radius must change over time. In order to formally proof it, consider
$$
\frac{d}{dt}\lVert\vec r(t)\rVert^2 = \frac{d}{dt}[\vec r(t)\cdot\vec r(t)]
= 2\vec r(t)\cdot \dot{\vec r}(t) = 0,
$$
since $\vec r\perp\vec v$. Hence, the distance is constant, meaning that we have a circular orbit. For the other direction see the above calculation for the ellipsis. $$\tag*{$\Box$}$$