How can we tell a theory is confining? Physically, I understand what it means for a theory to be confining. The elementary particles are not observable, but only composite particles are. The classic example is QCD, where quarks are confined to each other in $q\bar{q}$ pairs or $uud$, $udd$ etc. (this hasn't been shown analytically in theory, but is expected to happen experimentally and through lattice calculations).
I guess my confusion is: how is confinement formulated theoretically? What are people looking for when deciding whether a theory confines? Is there some vacuum expectation value which must be nonzero? Is it a statement about the spectrum of the Hilbert space? What are people calculating on the lattice?
 A: In my view, one of the best way to define what means "confinement" is to calculate Wilson loop. For instance, one can consider Euclidean compact abelian theory, 4D compact QED (it is defined on a lattice!) and calculate the Wilson loop in two different limits (strong coupling & weak coupling). Having performed the computation, you can see that in the limit of strong coupling, $g^2\gg 1$, Wilson loop $\propto\exp(-Af(1/g^2))$ where $A$ is area of Wilson loop. It is usually called "area law". In opposite limit, $g^2\ll 1$, you will see that loop $\propto\exp(-CPg^2)$, where $C$ is numerical constant and $P$ is perimeter of loop, so it is "perimeter law". Then, you can check that
$$\text{Wilson loop}=\mathcal{Z}[J]/\mathcal{Z}[0],$$
where $\mathcal{Z}$ is partition function of theory with current $J$. Then, it is not so hard to show that
$$\text{Wilson loop}\sim\exp(-V(R)),$$
where $V(R)$ is "quark" interaction potential. For $g^2\gg 1$ regime, $V(R)\propto R$, so infinite energy is required to separate two "quarks", "quarks" are confined. For $g^2\ll 1$, $V(R)\sim\text{const}$.
In conclusion, the theory is in confined phase if Wilson loop has "area law" behavior.
Moreover, I conjecture that explicit calculation of Wilson loop in continuum limit of 4D QED gives the same result. I have not made explicit calculation of Wilson loop for this theory but I believe that is possible, but it may be hard. I have seen a paper by Banks et. al where they present analytical calculations for some quantities in this theory and it seems that explicit calculation is possible.
Finally, I would like to emphasize that in my view it is useful to start from abelian theories, they are simpler and give intuition about confinement phenomenon.
You can find this topic in
Kogut, John B. "An introduction to lattice gauge theory and spin systems." Reviews of Modern Physics 51.4 (1979): 659, link
and also in Polyakov's book "Strings and gauge theories". Also, you can find "An introduction to confiment problem" by Jeff Greensite.
