This might be easiest to illustrate by example. Say your system consists of three point masses, located at positions $\vec{r}_1$, $\vec{r}_2$, and $\vec{r}_3$.
Now imagine we bring these three masses in from infinity, one at a time, to construct the system.
When moving the first mass from infinity to $\vec{r}_1$, the gravitational forces does no work on the mass, because there are no other masses to create a gravitational field.
When moving the second mass from infinity to $\vec{r}_2$, the gravitational force due to the first mass pulling on the second does work. Therefore, the final configuration of mass 1 at $\vec{r}_1$ and mass 2 at $\vec{r}_2$ has some energy, given by minus this work.
When moving the third mass from infinity to $\vec{r}_3$, the gravitational force of the first mass acting on the third and the second mass acting no the third both do work. Therefore, the final configuration of the three masses has gravitational energy, consisting of minus the work of the first acting on the second (from the previous paragraph), minus the work of the first mass acting on the third, minus the work of the second mass acting on the third.
This argument can be generalized to $N$ point masses.