What is the 't Hooft determinant? The 't Hooft vertex/determinant is somehow generated by instantons and is responsible for the generation of mass gap in pseudo-Goldstone bosons, such as an axion. 
For example, the complex Peccei-Quinn scalar couples to a fermion as $\phi\bar\psi_L\psi_R + h.c.$, which somehow develops a 't Hooft determinant when instantons come in the picture.

This then generates a linear term in the Higgs potential, explicitly breaking the $U(1)_{PQ}$ chiral symmetry, ultimately giving a mass to the axion.
As of this writing, there is nothing on Wikipedia on 't Hooft determinants. I also have been unsuccessful in finding a pedagogical introduction to such technology.
I have the following questions.


*

*What is a 't Hooft determinant?

*What is the role of instantons in this discussion?

*How does one compute modifications to the Higgs potential from a 't Hooft determinant such as the one shown in the diagram above?

 A: I am amazed that this question has not received an answer in the 4 years since it was asked. I'll give the broad picture of what the 't Hooft determinant is, and its relation to the axion story a la Peccei-Quinn,  providing original and pedagogical references.

't Hooft Determinant in QCD
We begin with the standard Euclidean Yang-Mills Lagrangian coupled to $N_f$ fermions.
$$S=\int d^4 x\, \left[\frac{1}{2g^2}F^2 - \sum_{f}^{N_f} \psi^{\dagger}_f \left(\gamma^\mu D_\mu + m_f\right) \psi_f + \mathcal{L}_{\textrm{g.f.}} + \mathcal{L}_{\textrm{ghost}}\right] \tag{1}$$
Here $\mathcal{L}_{\textrm{g.f.}}$ is the gauge-fixing term, and $\mathcal{L}_{\textrm{ghost}}$ is the Faddeev-Popov ghost Lagrangian. The corresponding path integral is:
$$Z = \int \mathcal{D}\psi^{\dagger} \mathcal{D}\psi \mathcal{D}A \mathcal{D} \bar c \mathcal{D} c \, e^{-S[A,\bar\psi,\psi, \bar c, c]}  \tag{2}$$
The basic idea of the 't Hooft determinant is to compute the path integral by first integrating over the gauge-field $A$ and ghost-fields $\bar c, c$, assuming the dominant contributions come from instanton configurations. This leaves us with an effective action for the fermions $S'[\psi^{\dagger}, \psi]$.
$$Z = C \int \mathcal{D} \psi^{\dagger} \mathcal{D}\psi \, e^{-S'[\bar \psi, \psi]} \tag{3}$$
The new effective action will of course involve non-trivial fermionic interactions. The hallmark nontrivial interaction that arises from integrating over the moduli space of a single-instanton configuration (using Gaussian approximation to handle leading quantum fluctuations, i.e. the semiclassical method) is the so-called 't Hooft determinant interaction [2]. This is a $2N_f$-fermion interaction term which can be written in the form of a determinant [3]. This determinant interaction is commonly denoted by $Y^{(\pm)}[\psi^{\dagger},\psi]$, where $+$ corresponds to the instanton and $-$ for the anti-instanton. Notice that it is an action-term, i.e. a functional in the fermion fields $\psi^{\dagger},\psi$. It is most conveniently expressed in momentum space:
$$Y^{(\pm)}\sim \int d^4x\,\,\, \left.\det\right._f J^{(\pm)}  \tag{4a}$$
$$J_{f_1 f_2} = \int \frac{d^4 k d^4 l}{(2\pi)^8} e^{i(k-l)\cdot x} \sqrt{M(k)M(l)}\,\psi^{\dagger}_{f_1}\frac{1\pm \gamma_5}{2}\psi_{f_2} \tag{4b}$$
Here, $f_{1,2}$ are flavor indices, and $\left.\det\right._f$ means determinant over flavor indices only. $M(k)$ is a specific function related to the Fourier transform of the fermionic zero mode. In fact for $N_f = 1$, we get a 2-fermion interaction which for a balanced collection of instantons and anti-instantons leads to a momentum-dependent mass $M(k)$! (hence the notation)
A possibly more illustrative way to view this interaction is as follows - for an instanton, the determinant interaction looks like the following.
$$\mathcal{L}_{\textrm{det}} \sim \det \begin{vmatrix}\bar\psi_1 P_L\psi_1 & \bar\psi_1 P_L\psi_2 & \cdots \\
\bar\psi_2 P_L\psi_1 & \bar\psi_2 P_L\psi_2 & \cdots \\
\vdots & \vdots & \ddots
\end{vmatrix} \tag{5}$$
where $\psi^{\dagger}_1 P_L \psi_2 = \psi_{1,R}\psi_{2,L}$. The anti-instanton configuration involves only right-handed spinor sources. Notice that the sum of the interactions due to an averaged instanton and anti-instanton produces an interaction term which preserves $SU(N_f)_L \times SU(N_f)_R$ and $U(1)_V$ flavor symmetries, but violates $U(1)_A$ flavor symmetry.
Keep in mind that, to calculate quantities in real QCD, we need not only effective interactions in a single instanton or anti-instanton, but rather in an entire ensemble of them. This is a the next logical step in the story, but is long and complicated. More details can be found in the second reference [3]. I will just mention that, a first-order approximation of the real QCD vacuum is a balanced ensemble of instantons and anti-instantons with instanton-interactions ignored, with fixed density $N/V\approx 1\,\,\textrm{fm}^{-4}$ and radius $\rho\approx 1/3\,\,\textrm{fm}$. In this case, the effective fermionic action will really just be the sum of $Y^{(+)}$ and $Y^{(-)}$ (with the kinetic terms of course), which perfectly provides a $U(1)_A$-violating term.

Relation to Axion in Peccei-Quinn Theory
To recap the entire point of the axion as per the original PQ paper [1], in normal QCD we have the strong-CP problem which is that the $\theta$ term is somehow identically zero.
$$\mathcal{L}_\theta = i\theta Q_5 \tag{6a} $$
$$Q_5 = \int d^4x\,\left(\partial^\mu j^5_{\mu}\right) = \frac{g^2}{16\pi^2} \int d^4x\, F_{\mu\nu} \tilde F^{\mu\nu} \tag{6b}$$
where $j^5_\mu(x)$ is the axial current, i.e. the Noether current arising from axial rotations. Recall that performing an axial-rotation of the fermion fields by $\psi\rightarrow \exp \left(i\eta \gamma^5\right)\psi$ induces a change in the effective action via
$$S\rightarrow S - 2 i \eta Q_5 \tag{7}$$
This shifts the $\theta$ parameter via
$$\theta\rightarrow \theta - 2\eta \tag{8}$$
If there are fermionic mass terms, this axial rotation will rotate the complex phase of the masses. The strong-CP problem is thus having simultaneously real fermion masses and $\theta = 0$.
In their original paper, Peccei and Quinn pointed out that we can find a natural resolution to the strong-CP problem by introducing a complex-scalar Higgs-type field that couples to the fermions via Yukawa interactions, which is charged under global $U(1)_A$ rotations.
$$\mathcal{L}_{\phi \psi \psi} = i\bar \psi \left[ G \phi \frac{1+\gamma_5}{2} + G^* \phi^* \frac{1-\gamma_5}{2}\right] \psi  \tag{9a}$$
$$\mathcal{L}_\phi = -\frac{1}{2}\left|\partial \phi\right|^2 - \mu^2 |\phi |^2 - h |\phi |^4  \tag{9b}$$
where $\mu^2 < 0$, and $G = |G|e^{i\gamma}$ is a complex number. I have expressed the general Yukawa interaction (9a) in terms of left and right-handed components, following the notation of the original PQ paper. Let us assume $\phi$ acquires a VEV $\langle \phi \rangle = \lambda e^{i\beta}$. It is clear that the fermions will acquire a mass proportional to $\lambda$ which will largely be fixed by the classical quartic potential in $\mathcal{L}_\phi$. Of course there may be quantum corrections to this potential, but we will assume those are small.
But what about the complex phase $\beta$? Notice that if we could somehow have $\beta = - \gamma - \theta$, a simple axial rotation $\psi\rightarrow \exp (i\theta\gamma_5 / 2 ) \psi$ would solve the strong-CP problem, i.e. $\theta$ will be cancelled as per (8), and the fermions will only have a real mass term as per (9). Spoiler alert: The complex phase $\beta$ is actually fixed to this magical value by instantons. This, I believe, is what you were looking for.
Allow me to briefly walk you through the original PQ paper [1]. I will reference equations in that paper by angle-brackets. Eq. <7> shows the full Lagrangian, with fermion, gauge, and axion fields. The next logical equation is eq. <10>, which is achieved by integrating out our femion and gauge fields $\bar\psi$, $\psi$, and $A$. Integrating the gauge fields $A$ is exactly the procedure outlined in the previous section of this answer, which led to the 't Hooft determinantal interaction for $\psi$. Integrating out the fermion fields is a monstrous task which I have not touched upon. The authors did not evaluate either of these integrals explicitly, but merely gave a schematic result <10> in terms of to-be-calculated functions $c_m^n(x,y)$. All that remains is an effective path integral for the complex-scalar field $\phi$ and $\phi^*$ (which has been reparameterized in terms of $\rho$ and $\sigma$ as per <9>).
The schematic evaluation they did is all that is necessary to see that VEV-phase $\beta$ will ultimately acquire the magical value $\beta = -\gamma -\theta$. A precise evaluation would be necessary to calculate quantum corrections to the VEV-magnitude $\lambda$. This is the rough idea for how the 't Hooft interactions would affect the VEV of $\phi$ and, in turn, affect the axion mass.
For now I will not dive into the calculation of the quantum-corrected axion mass, because I am not familiar with it at all, but I will point out some references which talk about it. The first (partial) computation of the quantum-corrections to the axion mass was done by Michał Spaliński in 1988 [4]. A more complete calculation was done recently by Giovanni Grilli di Cortona et al in 2016, see [5] and the references+discussions therein.

References
[1] Computation of the quantum effects due to a four-dimensional pseudoparticle; G. 't Hooft; Phys. Rev. D 18, 2199 (1978)
[2] Chiral Symmetry Breaking by Instantons; D. Diakonov; arXiv:hep-ph/9602375
[3] "CP Conservation in the Presence of Pseudoparticles"; R. D. Peccei and Helen R. Quinn; Phys. Rev. Lett. 38, 1440 (1977)
[4] "Chiral Corrections to the axion mass"; M. Spaliński; Zeitschrift für Physik C Particles and Fields volume 41, pages 87–90 (1988)
[5] "The QCD axion, precisely"; G. Grilli di Cortona et al; Journal of High Energy Physics volume 2016, Article number: 34 (2016); Arxiv Preprint: 1511.02867
