>Why does it accelerate linearly, if the only force I'm applying is a torque?

Forces and torques are separate things, but still related. A torque is *produced* by a force
$$\mathbf\tau=\mathbf r\times\mathbf F$$
So fundamentally you are just applying a force. That affects the linear motion of the object. If your force also happens to have a torque about the COM of the object, then the object will start spinning as well.

In other words, your force is not a torque. Your force has a torque about the COM. Torque is a manifestation of where and in what direction the force is being applied to the object.

Another way to look at this is to compare units. Forces have units of $\rm N$ where as torques have units of $\rm N\cdot\rm m$, so they cannot be the same thing.