According to Georgi's definitions in Section 2.1, a set of generators for a group $G$ is a set of linear operators $\{X_{a}\}$ in a representation $D$ of $G$ on a vector space $V$, such that for any smooth family $g(\alpha_{1}, \dotsc, \alpha_{n}) \in G$ we have
$$
D(g(\alpha_{1}, \dotsc, \alpha_{n}))
=
\exp\left(i\sum_{a}\alpha_{a}X_{a}\right)
$$
This means that each $X_{a}$ is a linear operator on a vector space $V$. The set of linear operators on some vector space is naturally a Lie algebra, where
$$
[X, Y]\vert v\rangle = XY \vert v\rangle - YX \vert v\rangle
$$
for all $\vert v\rangle \in V$. Equation (18.4) is concerned with a set of operators $R_{a}$ that generates a representation of $SU(2)$ and an operator $S$ that generates a representation of $U(1)$. So $R_{a}$ and $S$ are being treated as linear operators on the Hilbert space of states.
The operators
$$
u^{\dagger}
\quad d^{\dagger}
\quad e^{\dagger}
\quad \bar{\psi}_{r}^{\dagger}
\quad \bar{\ell}_{r}
\qquad r = 1, 2
$$
introduced in Georgi's equation (18.1) are creation operators for right-handed particles. A creation operator is a linear operator on the Hilbert space of states that adds particles of the corresponding kind. This means that $R_{a}$ and $u^{\dagger}$, for example, are both linear operators, and so we have a linear operator $[R_{a}, u^{\dagger}]$ that acts as
$$
[R_{a}, u^{\dagger}] \vert v \rangle
=
R_{a}u^{\dagger}\vert v \rangle
-
u^{\dagger}R^{a} \vert v \rangle
$$
on any state $\vert v \rangle$. The first equation in (18.4) says that $[R_{a}, u^{\dagger}]$ is the constant zero operator for any $a$. That is, we have
$$
R_{a}u^{\dagger} \vert v \rangle
=
u^{\dagger} R_{a} \vert v \rangle
$$
for all $\vert v \rangle$.
The creation operators $\bar{\psi}_{1}^{\dagger}$ and $\bar{\psi}_{2}^{\dagger}$ form what Georgi calls a "tensor operator" with respect to the representations of $SU(2) \times U(1)$. This concept is introduced in Chapter 4. A tensor operator is a set of operators that satisfies particular commutation relations with a set of generators. For example, the pair $\{\bar{\psi}_{1}^{\dagger}, \bar{\psi}_{2}^{\dagger}\}$ makes up a tensor operator with respect to the representation of $U(1)$ generated by the operator $S$, because it satisfies
$$
[S, \bar{\psi}_{r}^{\dagger}] = -\bar{\psi}_{r}^{\dagger}/6
$$
In other words, for any state $\vert v \rangle$ we have
$$
S\bar{\psi}_{1}^{\dagger}\vert v \rangle
-
\bar{\psi}_{1}^{\dagger}S\vert v \rangle
=
-\frac{1}{6}\bar{\psi}_{r}^{\dagger} \vert v \rangle
$$
To answer your first question, the Lie algebra generators $R_{a}$ and $S$ are linear operators on the Hilbert space of states, as are the operators $u^{\dagger}$, $d^{\dagger}$, $\dotsc$. The creation operators are linear operators on the Hilbert space carrying a representation of $SU(2) \times U(1)$, and they are elements of the Lie algebra of linear operators, but they are not generators of any interesting representation. The commutator in Equation (18.4) represents the commutator of linear operators, defined by $[X, Y] = XY - YX$. (I hope this answers your first question; I didn't quite understand the second and third sentences.)
To answer your second question, it seems to me that these equations are mathematically well-defined. They are equations between linear operators on a Hilbert space, with the left-hand side given by a commutator of two operators on that Hilbert space. For example, the action of $[R_{a}, u^{\dagger}]$ on any element of the Hilbert space is fixed by the actions of $R_{a}$ and $u^{\dagger}$, and the first equation requires that $[R_{a}, u^{\dagger}]\vert v \rangle = 0$ for all elements $\vert v \rangle$ of the Hilbert space of states. So it doesn't look like anything is ambiguous or contradictory.
