Some mathematics is unavoidable. First, in stating clearly what we mean by the curve being symmetrical.
Let us denote by $i^*$ the value of $i_1$ or $i_2$ for which $i_1=i_2$.
[It's easy to show that $i^*=\arcsin (n\ \sin \frac{A}{2})$] in which $A$ is the angle at the 'top' of the prism.]
If the curve were symmetrical we'd have the same angle, $D$, of deviation if$$i_1-i^*=i^*-i_2.$$[We know that $i_2$ is the other angle of incidence that will give the same D.]
The equation means that, if the curve is symmetrical, the mean of $i_1$ and $i_2$ is $i^*$.
This is not going to be the case because in order to relate $i_1$ to $i_2$ we have to apply the sine function, and the arcsine function when we apply Snell's law at each surface. Sine and arcsine are non-linear functions: you don't get the same size of change in $\sin \theta$ when $\theta$ changes from (say) 50° to 70° that you get when $\theta$ changes from (say) 50° to 30°. Similarly for arcsin.
I'm sorry if this is too mathematical, but I don't think we can explain the non-symmetry of the curve otherwise (except informally, by ray-sketching). The basic reason for the non-symmetry is the non-linearity of the sine function.
[It might be worth mentioning that the bottom of the curve is pretty symmetrical, that is for small variations of $i_1$ from $i^*$, we do have, approximately,$$i_1-i^*=i^*-i_2.$$I'm afraid the best way by far to explain why this should be the case is to appeal to mathematics, specifically to terms higher than the second in the relevant 'Taylor expansion' being negligible for small excursions from the minimum. Sorry!]