Angular momentum $\mathbf L$ is given by:
$$\mathbf L = \mathbf r \times \mathbf P$$
Now the torque $\mathbf {\tau} $ acting on an object is given by:
$$\boldsymbol {\tau } = \frac {d \mathbf L}{dt}$$
$$\Rightarrow \boldsymbol {\tau } = \frac {d (\mathbf r \times \mathbf P) }{dt}$$
$$\Rightarrow \boldsymbol {\tau } = \mathbf r \times \mathbf F + \mathbf v \times \mathbf P$$
Since $\mathbf v $ and $\mathbf P$ are in the same direction therefore
$$\mathbf v \times \mathbf P = 0$$
i.e.,
$$\Rightarrow \boldsymbol {\tau } = \mathbf r \times \mathbf F $$
Now as you correctly say that $\mathbf L$ is constant therefore
$$\boldsymbol {\tau } = \mathbf r \times \mathbf F =0$$
Clearly since $\mathbf r \neq 0$ then this means that
$\mathbf F$ at all instants points towards the centre (i.e., to say that line of action of the force passes through the centre).
Note that force cannot be zero as a centripetal force is always required for maintaining circular motion.
Now as you can see from the figure during transition from outer orbit to inner orbit the force and velocity vector have acute angle between them therefore the force acts to accelerate the object and hence increase it's speed.
Also it should be noted that for angular momentum to be conserved the absence of force isn't a necessary criteria (example elliptical orbits of planets).
Regarding the comment:
So if I calculated the speed of particle after the radius has been reduced down to $0.5$ it would turn out to be $2v$, where $v$ is the original speed? Is there a formula that I can use?
Yes as you know that
$$\mathbf L = \mathbf r \times \mathbf P$$
$$\frac {\mathbf L}{m} = \mathbf r \times \mathbf v \tag 1$$
Clearly after the radius reduces to $\mathbf {r'}$ then the velocity becomes $\mathbf {v'}$
$$\Rightarrow \frac {\mathbf L}{m} = \mathbf {r'} \times \mathbf {v'} \tag 2$$
Therefore from $(1)$ and $(2)$ (also noting that after the particle comes into orbit the vectors have $90°$ angle between them)
$$\Rightarrow v'r'=vr$$
Then
$$v'= v \frac {r}{r'}$$
Note that this has been explained in the following Vsauce video (explanation starts from 10:05 to 13:15):