At the end of the day, all these quantities are just definitions. So the only thing one can do is show that these definitions are natural.
First I will motivate why angular momentum is important. Noethers' theorem says that for every symmetry of equations of motions there is some conserved quantity. Newtons' equations of motions are translationally invariant and this turns out be equivalent to momentum being conserved. This makes momentum an important quantity.
Similarly, Newtons' laws of motion are invariant under rotations and this leads to angular momentum, $\vec L=\vec r\times\vec p$, to be conserved. It is also natural to study the time derivative of this quantity, the torque, similar to how we study forces as the time derivative of momentum.
Next, we will see that based on parallels with normal momentum it is natural to define $I=mr^2$. We can split regular momentum into mass and velocity. Velocity is the time derivative of the position $\vec r$, which I will call the 'dynamical variable'.
$$\vec p=\overbrace{m}^{\text{inertia}}\underbrace{\vec v}_{\text{dynamical}\\\text{ variable}}$$
Angular momentum is defined as $\vec L=\vec r\times\vec p$. Angular momentum is associated with rotations, so it is natural to look at the angle as the dynamical variable. If we assume circular motion for a second we can take the momentum to be perpendicular to $\vec r$.
\begin{align}
L&=rp\\
&=mrv
\end{align}
We can also write the velocity as $v=\omega r$, where $\omega=\dot\theta$. This gives us a similar split in an inertial term and a time derivative of the dynamical variable $\theta$.
$$
\vec L=\overbrace{mr^2}^{\text{inertia}}\underbrace{\omega}_{\text{dynamical}\\\text{ variable}}$$
This last equation can be interpreted as follows: for something which has high $mr^2$, it is hard to change the rotation rate. We can argue that an $I=mr$ says the same, but because of the equations of motion $I=mr^2$ is more natural.
We can see the factor $r$ as a conversion factor for going from angles to displacements. Similar to how $s=r\theta$, where $s$ the arc length. In the torque equation $\tau=I\alpha$ we get one factor of $r$ for converting from angular acceleration to acceleration and we get one factor for converting from torque to force. See the following (very) handwavy explanation.
\begin{align}
\tau&=I\alpha\\
rF&=(mr^2)(\frac{a}{r})\\
F&=ma
\end{align}