Topological twists of SUSY gauge theory Consider $N=4$ super-symmetric gauge theory in 4 dimensions with gauge group $G$. As is explained in the beginning of the paper of Kapustin and Witten on geometric Langlands, this
theory has 3 different topological twists. One was studied a lot during the 1990's and leads
mathematically to Donaldson theory, another one was studied by Kapustin and Witten (and mathematically it is related to geometric Langlands). My question is this: has anyone studied the 3rd twist? Is it possible to say anything about the corresponding topological field
theory?
 A: Some details of the third twist can be found in section 6 of Ref. 1. The BPS equations correspond to a non-abelian version of the monopole equations considered by Witten in Ref. 2. Some aspects of this topological field theory were considered in Ref. 3, generalising the analysis in Ref. 2 to the non-abelian case.
In each of the three topological twists of $N=4$ supersymmetric Yang-Mills in 4d, the set of bosonic fields contains a gauge field and two real scalars (just as in the twist of $N=2$ supersymmetric Yang-Mills that gives Donaldson-Witten theory). In the respective twists, the remaining four bosonic degrees of freedom in the $N=4$ supermultiplet assemble into either (i) a scalar and a self-dual two-form, (ii) a one-form, (iii) two chiral spinors. (Of course, all fields are valued in the adjoint representation of the gauge group.) Twist (i) gives the Vafa-Witten theory of Ref. 4. Twist (ii) is the one first noted by Yamron in Phys. Lett. B213 (1988) 325-330, considered by Marcus in Ref. 5, and more recently by Kapustin and Witten in the context of geometric Langlands. Twist (iii) is the one mentioned in the paragraph above.
On a compact Kähler four-manifold $X$ with $b_2^+ (X) > 1$, I believe that the close analogy between twists (i), (iii) and Donaldson-Witten theory relies on a vanishing theorem similar to that used in section 3 of Ref. 2 in the abelian case of twist (iii). The implication being that all solutions of the BPS equations resulting from twists (i) and (iii) correspond to instantons on $X$ (with the four twisted scalars equal to zero).
Twist (ii) is a bit more subtle in the sense that it actually gives rise to a family of topological field theories, with each member labelled by a point on ${\mathbb{CP}}^1$. This is so because, up to an irrelevant overall scale, one can define a topological BRST operator from any complex linear combination of the two scalar supercharges which survive this twist. To quote Witten; "there are no trivial equivalences among this family of topological field theories, only interesting equivalences that come from dualities". In certain special cases, solutions of the BPS equations can be thought of as flat complexified connections of the gauge bundle (e.g. as in Ref. 5) rather than instantons.

References:

*

*C. Lozano, Duality in Topological Quantum Field Theories, arXiv:hep-th/9907123.

*E. Witten, "Monopoles and four-manifolds", arXiv:hep-th/9411102, Math. Res. Lett. 1 (1994) 769-796.

*J. M. F. Labastida & M. Mariño, "Non-abelian monopoles on four-manifolds", Nucl. Phys. B 448 (1995) 373-398, arXiv:hep-th/9504010.

*C. Vafa, E. Witten, "A strong coupling test of $S$-duality", Nucl. Phys. B 431 (1994) 3-77, arXiv:hep-th/9408074.

*N. Marcus, "The other topological twisting of $N = 4$ Yang-Mills", Nucl. Phys. B 452 (1995) 331-345, arXiv:hep-th/9506002.

A: The Kapustin-Witten paper

https://arxiv.org/abs/hep-th/0604151

says (on page 17) that two of the three twists are related to Donaldson theory:

Two of the twisted theories, including one that was investigated in detail in [45: Vafa Witten], are closely analogous to Donaldson theory in the sense that they lead to instanton invariants which, like the Donaldson invariants of four-manifolds, can be expressed in terms of the Seiberg-Witten invariants

By Vafa-Witten, I mean

https://arxiv.org/abs/hep-th/9408074

The least studied twist among the three was studied by Neil Marcus

https://arxiv.org/abs/hep-th/9506002

but I am not sure whether everyone in that field thinks that the paper is right.
