Why does it accelerate linearly, if the only force I'm applying is a torque?
Forces and torques are separate things, but still related. A torque is produced by a force $$\mathbf\tau=\mathbf r\times\mathbf F$$ So fundamentally you are just applying a force. That affects the linear motion of the object. If your force also happens to have a torque about the COM of the object, then the object will start spinning as well.
In other words, your force is not a torque. Your force has a torque about the COM. Torque is a manifestation of where and in what direction the force is being applied to the object.
Another way to look at this is to compare units. Forces have units of $\rm N$ where as torques have units of $\rm N\cdot\rm m$, so they cannot be the same thing.
Now, getting to how a tangential force causes a linear acceleration, it is easier to go more general. Let's say we have a rigid body of total mass $M$ consisting of $N$ particles. We know that the center of mass of the body is given by $$r_{cm}=\sum_i^N\frac{m_i\mathbf r_i}{M}$$ Taking time derivatives, we can write the velocity and acceleration of the center of mass as well $$v_{cm}=\frac{\text d}{\text dt}r_{cm}=\sum_i^N\frac{m_i\mathbf v_i}{M}$$ $$a_{cm}=\frac{\text d}{\text dt}v_{cm}=\sum_i^N\frac{m_i\mathbf a_i}{M}$$
Using Newton's second law we know that, for each particle $$\mathbf F_i=m_i\mathbf a_i$$ and so putting it all togther we see that $$a_{cm}=\sum_i^N\frac{\mathbf F_i}{M}$$ or rewriting things $$\sum_i^N\mathbf F_i=Ma_{cm}$$
Now we need to think about the force we apply and the forces on each particle. The net force on any particle is the sum of any external forces acting on that particle and any forces between the other particles in the body: $$\mathbf F_i=\sum_{ext}\mathbf F_i^{ext}+\sum_{j}^N\mathbf F_{j\rightarrow i}$$
Therefore: $$\sum_i^N\mathbf F_i=\sum_i^N\sum_{ext}\mathbf F_i^{ext}+\sum_i^N\sum_j^N\mathbf F_{j\rightarrow i}$$
Now, this double sum in the second term covers all $(i,j)$ pairs, so we can rewrite this term by double counting and then correcting for our double counting:
$$\sum_i^N\mathbf F_i=\sum_i^N\sum_{ext}\mathbf F_i^{ext}+\frac12\sum_i^N\sum_j^N\mathbf (\mathbf F_{j\rightarrow i}+\mathbf F_{i\rightarrow j})$$
However, we know that by Newton's third law, $\mathbf F_{j\rightarrow i}=-\mathbf F_{i\rightarrow j}$ So our "double count" sum is actually $0$. Therefore: $$\sum_i^N\mathbf F_i=\sum_i^N\sum_{ext}\mathbf F_i^{ext}=\mathbf F_{net}^{ext}$$
Therefore, bringing everything together: $$\sum_i^N\mathbf F_i=\mathbf F_{net}^{ext}=Ma_{cm}$$
Notice how we have not assumed anything about the torques of these forces! So we see that, independent of what torques the applied forces might have, external forces alter the linear acceleration of the center of mass of the system of particles.