Angular momentum(and torque) does not necessarily mean that there are rotations in place. you can define angular momentum for straight lines, however, that isn't very useful to solve any problem.
If you have a particle moving in a straight line and apply a force perpendicular to it, it won't rotate unless there is a centripetal force somewhere. In your exemple, if you only apply a perpendicular force, the particle will simply move in both x and y directions in a parabola.
What will happen is that, say, you have a particle moving in a straight line in the x direction. The angular momentum is: $r \times p_x $. Then you apply the perpendicular force, adding a y component into the velocity. Since the force is perpendicular only, then the linear momentum is conserved in the x direction at all times, thus
$$ \frac{dp_x}{dt} = 0$$
The angular momentum while applying the force is, thus: $r \times (p_x + p_y) $. Where $p_x$ is constant and $p_y$ is varying due to the force, the torque is: $r \times \frac{dp_y}{dt}) $
Once the torque vanishes, no forces act in the y direction, thus
$$ \frac{dp_y}{dt} = 0$$
And then the particle continues in a straight line.Take the time derivative of angular momentum after the torque vanishes:
$$ \frac{dL}{dt} = v \times (p_x + p_y) + r \times (\frac{dp_y}{dt} + \frac{dp_y}{dt}) $$
The first term vanishes because the velocity vector is always parallel to the linear momentum, then you are left with the second, but both time derivatives are zero, thus:
$$ \frac{dL}{dt} = 0 $$
What does this mean? Nothing, because there is no rotations, however it's still a true result to say that angular momentum is conserved in absence of torques.