Do force fields come from potential fields, or do potential come from forces?

Please excuse me if this question is a duplicate. I tried my best but I didn't find an existing question for this.

In physics class, I was first introduced to gravity in terms of a force, specifically, the inverse square law. At the same time, there is also the concept of gravitational potential energy, and the gravitational force vector always pointed from high potential to low potential.

Mathematically, you could describe that as $F_g = -\nabla g$, where $g$ is the gravitational potential field. On the other hand, you could also say in terms of work $g = -\int F_g \cdot dx$ (assuming a point at $\infty$ has 0 potential).

The way I typically thought about it is that something like flux as the "root source" of the fields, and the flux comes from a property such as mass for gravity or charge for electromagnetism. Then, the potentials are a result of the force. That makes sense to me since the force field is like the flux area density, which would be proportional to the inverse square of the distance from the source of the flux. For example, the electric force weakens by a factor of 4 when your distance doubles, since the flux is being spread over a Gaussian surface that's 4 times larger.

My question is, is the flux responsible for setting up the forces, from which the potentials come as a result, or does it set up the potentials, and the forces simply act along the gradient? Or, are these interpretations the same?

In problems I usually thought about it in terms of the differential way but I would love to get some insight on this. Thanks.