Does the inverse square law have anything to do with conservative behavior of the central forces?
Not really, no. A field is conservative if it can be written as the gradient of a potential function:
You could equally well define the potentials $f=kr^2$, or $f=kr^5$, etc., which will produce vector fields that are respectively linear and quartic with $r$.
All central forces (where central means spherically symmetric, that is independent of angular variables) are conservative: see the quesiton Are all central forces conservative? Wikipedia must be wrong.