Is there a mathematical derivation of the inverse square law that doesn't depend on geometry or empirical data fitting?
What do you mean by "doesn't depend on geometry?". If you are referring to the Coulomb law for the electric field generated by a point charge, it can be derived from Maxwell's equations. These have their foundations in the symmetry principles of the special theory of relativity, but as fundamental laws of nature, they can only be justified by the experience.
The Gauss' Law (1st Maxwell equation in integral form) gives
$$\iint_S \mathbf E \cdot ~\mathrm d\mathbf a = 4\pi\iiint_V \rho ~\mathrm dV$$
where $S$ is a closed surface that contains the charges, $V$ is the volume enclosed by such surface and $\rho$ is the density of electric charge. For a point charge at rest, let's take $S$ to be a sphere of radius $R$ centered in the charge. From symmetry arguments it is clear that $\bf E$ is constant on the surface of the sphere and it is perpendicular to it. Its modulus will depend only on the radius $R$, i.e. the distance from the charge. In this special case the left hand side of the equation is
$$ E(R)\iint_S\,\mathrm d\mathbf a = 4 \pi R^2 E(R)$$
The right hand side is just the total charge contained in the sphere (times $4\pi$) and so we have in the end
$$4 \pi R^2 ~ E(R)=4\pi e$$
that gives the Coulomb law
Identical considerations can be used to derive the inverse-square law for the gravitational force.
Here is an incredibly simple derivation of the the inverse square law for gravity which shows how it must rely on geometry..
A simple way to think about the gravitational field of an object is to imagine a fixed number of "lines of force" that radiate from the object evenly into space.
Let's suppose the number of lines of force produced by an object is directly proportional to its mass, so..
where n is the number of lines of force produced by the mass m and k is a constant.
Now assume the density of the lines at any given point in space represents the strength of the gravitational field at that point. So at a distance r from the object the density of the lines of force is..
n/(surface area of the sphere of radius r)
which is n/(4*pir^2)=km/(4*pir^2)=Gm/(r^2)
where G=k/4*pi is a constant.