I just read the first chapter of Becker-Becker-Schwarz. To quote:

A remarkable discovery made in the late 1990s is the exact equivalence (or duality) of conformally invariant quantum field theories and superstring theory or M-theory in special space-time geometries.

Can this AdS/CFT duality be used to work in string theory or M-theory instead of quantum Yang-Mills theory, make the corresponding theory rigorous, and then go back to quantum Yang-Mills theory, thereby solving the open Millenium Prize question?

-
Mapping the problem to string theory (if that could be done) would not in itself be an increase in rigor, since string theory uses the same QFT concepts that lack rigorous mathematical formulation. (btw apparently the work of Fields Medalist Hairer will help with the latter.) But mapping it to string theory might be a source of ideas or intuitions which could then be inspiration for a proof. – Mitchell Porter Sep 2 '14 at 10:28

In principle yes, but there are several conceptual and technical issues that make it unclear how this could be achieved. Even though the AdS/CFT correspondence is conjectured to be exact(with much evidence hinting at this), it is hard to prove this essentially because in order to do calculations, one still has to use approximations and perturbation theory on one side or the other. This is essentially due to its dual nature: it relates a strongly coupled theory on the one side to a weakly coupled one on the other.

This can be understood much better in terms of the fundamental coupling constants of the theory: the string coupling is given by $g_s$ and the Yang-Mills coupling by $g_{YM}$, they are related by $g_{YM}^2=4\pi g_s$.

On the gravity/string side, calculations are feasible (unless you want to solve string theory on curved backgrounds, which is essentially an open problem) in the supergravity approximation of the theory. This is valid if we take $g_s\rightarrow0$ and also assume that the degree of the gauge group (the number of colours) $N$ is taken to infinity. Their product, the 't Hooft coupling $\lambda=g_{YM}^2N=4\pi g_sN$ however must be held fixed but much larger than one. For the Yang-Mills theory, perturbation theory is valid just on the other end of the parameter range, namely when $\lambda$ is very small.

This is useful for many calculations, since it allows one to chart the nonperturbative region of one theory by looking at the perturbative region of the other, but it makes it hard to prove anything in a rigorous way, especially within the axiomatic approach required for the solution of the Millenium problem. The requirement of large $N$ further restricts the applicability of the duality with respect to proving something for the generic gauge group $SU(N)$.

Apart from this, as mentioned in the other answer, the original and most well-known formulation of is, as the name suggests, between string theory and a conformal field theory, which by definition has no mass gap. In order to adress issues like confinement and the associated mass gap, one would have to work in other versions of the duality, e.g. the Witten model, which is $AdS_7\times S^4$, with a compactified dimension on the $AdS$ part, which breaks supersymmetry and introduces a mass scale.

-
Could you point me to a reference for the Witten version of the AdS/CFT duality? – Aman Chawla Sep 2 '14 at 11:13
Here you go: arxiv.org/abs/hep-th/9803131 . It is actually a very important reference, with over 2000 citations. – Frederic Brünner Sep 2 '14 at 11:38
It seems that there is a paper that already takes an approach similar to the one I suggested: arxiv.org/abs/hep-th/0402207 – Aman Chawla Sep 7 '14 at 1:15
How is that paper related to the Millennium problem? – Mitchell Porter Sep 7 '14 at 6:48

To be clear what we're talking about (as I'm not totally sure this is what the question intended), I'll talk about the paradigmatic example of AdS/CFT, the equivalence between $\mathcal N=4$ Yang-Mills on the one hand, and IIB string on (asymptotically) $AdS_5\times S^5$ on the other (at general parameters: no t'Hooft limits etc).

We are very much closer to defining rigorously the left hand side of this correspondence than the right. What I have in mind here is putting the theory on a lattice, and taking some appropriate limit as the lattice spacing becomes small. The difficulty is in proving that this works, and gives all the properties that you'd expect (the right symmetries must be restored in the limit for example, since the lattice will break some of these). On the other side, I think it's fair to say that there is not a clear idea of what a formulation of nonperturbative string theory would even look like.

Indeed, it's quite common to hear something like "Take your favourite theory of quantum gravity, for example N=4 Yang-Mills..." with the implication that it is the 4d field theory that defines the bulk string theory, rather than the other way round as you'd like to suggest. The challenge here is to repackage the CFT variables to that what you get bears some resemblance to gravity...

As a final point, in reference to the Millennium Prize, is that $\mathcal N=4$ is qualitatively different from ordinary Yang-Mills, in particular being conformal, so there is no mass gap. Integrability gives some chance to `solve' $\mathcal N=4$, without reference to $AdS$, but this would not bag you a million dollars (though it may lead to some insight in how to proceed).

-