Conservation of energy while swinging on a swing without slack So I have an issue with the following classical problem:

Let's say you are on a swing of length $l$ hanging downwards, and you get pushed in a horizontal direction with speed $v_0$. How large should this speed be so that you'll swing all over the top without the chain ever becoming slack?

A "solution" that I know of is to consider that the swing has no slack if $-\cos(\theta) < l \dot{\theta^2} / g $, and due to conservation of energy $m\dot{r}_0^2 / 2 = 2mgl + m\dot{r}_t^2 / 2$, from which $v_0 > \sqrt{5gl}$ follows.
However, it is entirely unclear to me why are we able to use the law of conservation of energy here. Clearly, some force due to the swing acts on the object at its end (otherwise the object wouldn't go up in the first place). In my opinion, if we don't assume that the swing is a "spring" and has its own potential energy, the energy in such a system is not (in general) conserved. In fact, we have ($r$ - position vector, $V$ - the potential due to gravity, $N$ - the force of the spring, $k$ - unit vector in the direction opposite gravity):
\begin{equation}
m \ddot{r} = - \nabla{V} + N \\
\frac{\partial}{\partial t} (m \dot{r}^2 / 2) = m \dot{r} \ddot{r}  = - \dot{r} (\nabla V + N) \\
m\dot{r}^2 / 2 + mgr \cdot k = \int_{t_0}^{t} N(r(\tau)) \frac{\partial r}{\partial \tau} d\tau
\end{equation}
where the integral is an already parametrized curve integral over the trajectory of the object. What am I missing here?
 A: 
Clearly, some force due to the swing acts on the object at its end (otherwise the object wouldn't go up in the first place).

Yes, but that force does no work since at each point the force is perpendicular to the velocity. In this problem the only force which has a component that is not perpendicular to the velocity is gravity. That force is associated with the gravitational potential energy.

In my opinion, if we don't assume that the swing is a "spring" and has its own potential energy, the energy in such a system is not (in general) conserved.

While you certainly can assume that the swing is a spring with its own elastic potential energy, it is not necessary. Without it the energy is still conserved. If you do assume the swing is a spring then you will find that the force is not always perpendicular to the velocity. It is during those times that energy is going into or out of the elastic potential energy. But at the moments when the force is perpendicular to the velocity then the amount of elastic potential energy is constant.
A: If you perform this experiment in a room-sized vacuum chamber, energy is ideally conserved.
The loss of energy in an open environment will be due to air friction, there will be a drag force in the direction of the velocity vector
$$F_D=\frac{1}{2} \rho v^2 C_D A$$
$\begin{align*} 
where \; &F_D = drag\:force \\ & \rho \;\;\: = density\:of\:the\:fluid \\ &v\:\;\;= speed \:of\:the\:object\:relative\: the\:fluid \\ & C_D=drag\: coefficient \\ & A \;\;= area\:of\: cross\:section\: of\:the\:object
\end{align*}$
The drag coefficient for a sphere in the air is 0.5. Moreover $F_D 
\propto v^2 $, for small v; the drag coefficient is negligible, energy loss due to drag force can be ignored and energy conservation can be applied.
If you want to calculate taking into account the energy loss due to drag force, you can integrate the drag force over the path taken by the object and subtract the integral it from the left side of your last equation.
