Existing answers offer some good physical insight but they are missing an essential ingredient I think, and that is to discuss entropy.
Many physical processes are the joint result of considerations of energy and entropy. For an isolated system (one that is not undergoing interactions with other systems), the system's internal energy stays constant and its entropy increases over time until it reaches some maximum value consistent with the other properties such as the fixed amount of energy.
In the case of a crystal what happens is that some process has acted to remove energy from the system, with the result that the atoms do not have enough energy to escape from their mutual attraction, so they have to gather close to one another. The puzzle now is why they don't just gather in some more haphazard way, rather than a regular lattice. To understand that you have to consider both the position and the motion of the atoms---their momentum.
Entropy, in physics, quantifies how many ways a given system can be rearranged internally without changing its overall properties, and this can be regarded as a measure of irregularity of lack of structure. Crystals have a regular arrangement of atoms, so this suggests the entropy is low, which is surprising. But in fact to maximise the entropy the system must maximise the range of available states of both position and momentum.
The states of lowest potential energy are the ones where the atoms form a regular lattice.
When the atoms go to those states, they make it possible for there to be more kinetic energy in their vibrations, and therefore a greater range of momentum states.
Overall this can result in more irregularity (i.e. entropy) in the complete state of position and momentum, compared to the case where the locations are less regular and the vibration (and consequently the range of momentum) is smaller.
Mostly when people discuss this they simply say that the system "seeks" or "goes to" the state of lowest energy. In fact the system cannot lower its energy overall, if it is an isolated system, so you should not accept such accounts. What people are really saying is that the system tends to lower its potential energy, but to understand why this is so we need to think about the kinetic energy too, and the way it impacts on the entropy, as I have discussed.
The regularity of the crystal reflects the fact that the atoms (or, more generally, the molecules) are all alike, so the minimum potential energy arrangement of one group is the same as for another group. So one can expect the least potential energy when the whole crystal is perfect. Crystals in general are not perfect (they have grains and boundaries) because the atoms originally gathered in multiple places at once, and these places just meet each other randomly. It takes a local input of energy to reorient any given grain. This
can happen but it takes a long time if one is just waiting for it to happen
by a random concentration of energy, so the crystal remains imperfect.
In the case of a system which is not isolated, it is the joint entropy of the system and its environment which is maximised. In this case, when a crystal forms it passes energy to its environment, with the result that whereas the crystal gets a state of lower entropy, the increase of entropy in the environment more than compensates. Notice that this principle extends to a vast array of phenomena. It is very common find a process where the entropy in one part of the world became smaller, because this led to an overall increase of entropy in the wider world. Such effects are at the root of most of chemistry and biology.