How does the BRST transformation act on ghost fields? I understand the general idea behind constructing the BRST symmetry: take a generic gauge transformation
$$\begin{equation}
e^\omega,
\end{equation}\tag{1}$$
where $\omega$ is Lie-algebra valued, and replace $\omega$ with some Grassmann number $\epsilon$ times the ghost field $c$:
$$\begin{equation}
e^\omega\rightarrow e^{\epsilon c}.
\end{equation}\tag{2}$$
Of course this doesn't tell you how the ghosts or Nakanishi-Lautrup fields ($B$) change. I've seen an additional term added, which is something like:
$$\begin{equation}
\omega\rightarrow \epsilon c + \epsilon~\text{Tr}(c^2b),
\end{equation}\tag{3}$$
but I can't quite work out how this term comes into play. I assume that: one, the extra term is fine because it goes as a BRST variation of the ghosts $\delta c\sim c^2$ and must be exact, and two, that the extra term somehow imposes conditions on the transformations for the antighost ($b$) and $B$ fields. I could try brute-forcing this calculation, but I can't really figure out how the Slavnov operator (or really gauge transformations in general) act on ghosts, antighosts or auxiliary fields. Specifically, gauge transformations act by conjugation (+ a derivative) on gauge bosons, since they are in the adjoint representation. Of course, so are $c$, $b$, and $B$, but acting naively on $c$ like $$c\rightarrow c'=g(c-Dg)g^{-1}\tag{4}$$ seems to only produce nonsense. Can someone help clarify this behavior?
 A: TL;DR: The BRST transformation rules for the ghost fields follow from requiring that the BRST/Slavnov operator squares to zero (= is nilpotent in physics jargon).
More details: When studying the BRST formalism, one should be aware that there exists$^1$ at least 2 versions relevant to OP's question:

*

*The Hamiltonian Batalin–Fradkin–Vilkovisky (BFV) formalism. Assuming no second-class constraints, and that the first-class constraints $G_a(q,p)$ are irreducible,

*

*the minimal field multiplet is the original position variables $q^i$ and the Faddeev-Popov (FP) ghost ${\cal C}^a$,


*while the non-minimal field multiplet is the Lagrange multiplier $\lambda^a$ and the FP antighost $\bar{\cal C}_a$.
On top of that there are the corresponding momenta, namely $p_i$, the FP ghost momenta $\bar{\cal P}_a$; the Nakanishi-Lautrup (NL) field $B_a$, and the FP antighost momenta ${\cal P}^a$, respectively.
As usual, the above canonical pairs of operators can be realized via a suitable Schrödinger representation.
Note that the bar/overline in $\bar{\cal C}_a$ and $\bar{\cal P}_a$ does not denote complex nor Hermitian conjugation. Rather, it signifies a negative ghost number.
Also note that that the FP ghost ${\cal C}^a$ and antighost $\bar{\cal C}_a$ are not a canonical pair, i.e. they super-commute.
The BRST charge operator
$$\begin{align} Q~=~&Q_{\min} + Q_{\text{non-min}},\cr
Q_{\min}~=~&G_a {\cal C}^a +\frac{1}{2}\bar{\cal P}_cf^c{}_{ab}{\cal C}^b{\cal C}^a +\ldots, \cr
Q_{\text{non-min}}~=~&B_a{\cal P}^a, \end{align}$$
is (among other things) a Grassmann-odd operator of ghost number +1 that squares to zero
$$ Q^2~=~0,$$
cf. Ref. 1. The above ellipses $\ldots$ refers to higher-order terms, where each term contains one more ghost ${\cal C}^a$ than ghost momenta $\bar{\cal P}_a$. Such term may be needed if the structure functions $f^c{}_{ab}(q,p)$ are not constant.
See also e.g. this related Phys.SE post.
The BRST transformations $$\delta~=~[Q,\cdot]$$ are given by the super-commutator. So e.g. the ghost transforms as
$$\delta {\cal C}^c~=~[Q,{\cal C}^c]~=~\frac{1}{2}f^c{}_{ab}{\cal C}^b{\cal C}^a+\ldots,$$
cf. OP's title question.


*The same formalism where half of the variables (mostly momenta) have been integrated out. The remaining variables are the original position variables $q^i$, the FP ghost ${\cal C}^a$, the FP antighost $\bar{\cal C}_a$, and the NL auxiliary field $B_a$. This is the version usually found in textbooks.
The field $b$ found in the string theory literature is either the antighost $\bar{\cal C}$ or the ghost momentum $\bar{\cal P}$, depending on context.
References:

*

*M. Henneaux & C. Teitelboim, Quantization of Gauge Systems, 1994; Chapter 9.

--
$^1$ There is also a Lagrangian BV formalism.
