First, to be clear about a few things:
- The spontaneously broken symmetry leading to superconductivity (no resistance, Meissner effect etc.) is $U(1)$-phase rotation symmetry, associated with conservation of (Cooper pair) particle number. Other symmetries, the crystal space group and spin-rotations, can be additionally broken. Your question pertains these other symmetries.
- The gap function $\Delta(\bf{k})$ by itself is a complex scalar function, i.e. an element of $\mathbb{C}$ for each $\mathbf{k}$. However, the transformation properties under space group and spin-rotation transformations allows the identification of additional structure. Most experiments measure the amplitude of this complex quantity.
- The total wavefunction must be antisymmetric. Therefore, if the spin-part is antisymmetric (singlet pairing) the orbital (space-group dependent part) must be symmetric, so even under parity. If the spin-part is symmetric (triplet pairing), the orbital part must be odd under parity.
- A parity/space inversion transformation simply replaces $\mathbf{k} \to -\mathbf{k}$.
Then, to answer your questions.
The scalar/vector denomination is about spin-rotation transformations.
A basis of states of the fermions is given by $|k,\sigma>$, where $k$ is the momentum and $\sigma = \uparrow,\downarrow$ the spin. The complex scalar function above can be decomposed as
$\Delta(\mathbf{k}) = \sum_{\sigma,\sigma'} < k,\sigma | \Delta_{\mathbf{k},\sigma\sigma'} | k,\sigma' >$ (Eq.1)
Here $\Delta_{\mathbf{k},\sigma\sigma'}$ (using the notation of the slides in your question) is a 2$\times$2-matrix for each point $\mathbf{k}$.
We can write out the 2$\times$2-matrix in the singlet-triplet basis. The singlet-state is one-dimensional, as such there is one degree of freedom to be specified by the $\mathbf{k}$-dependence. One can then call this the scalar wavefunction $\psi(\mathbf{k})$ (as in the slides), since the spin-state is already defined. The spin-singlet is antisymmetric, so the momentum-dependent part must be symmetric (even under parity).
Conversely, the triplet-basis is three-dimensional, meaning there are three degrees of freedom to be specified. Collecting the Pauli matrices in a vector $\vec{\sigma} = (\sigma^x, \sigma^y, \sigma^z)$, we can write the momentum dependence as $\vec{d}(\mathbf{k}) \cdot \vec{\sigma}$, which is a superposition of 2x2-matrices acting on the spin-subspace for each momentum $\mathbf{k}$. One can therefore call this a vector wavefunction $\vec{d}(\mathbf{k})$. It transforms as a vector under spin-rotations. Nevertheless, the gap function in (Eq.1) is a complex scalar.
In fact, these are pseudoscalar and pseudovector functions, meaning they are even under parity transformations. The wavevector $\mathbf{k}$ is of course odd under parity.
If $g$ is an element of the transformation group, then the action on the $d$-vector is specified as
$g \rightharpoonup \vec{d}(\mathbf{k}) = R_\mathrm{s}(g) \vec{d}(R_\mathrm{o}\mathbf{k})$
Here $R_\mathrm{o}$ is the orbital representation and $R_\mathrm{s}$ the spin representation, which are odd and even under parity respectively. This notation is from slide 27 of this presentation by the same author as in your link.
To understand how the representative functions, like $k_x \hat{x} - k_y \hat{y}$, transform, recognize that the unit vectors are basis vectors of the $d$-vector, and therefore transform as pseudovectors, while the $k$-components transforms as vectors.
In Kaba and Sénéchal Table I, you can find explicit matrices for $R_\mathrm{o}$ (called $U$) and $R_\mathrm{s}$ (called $R)$. In particular, the transformation under $\sigma_h$ (called $\sigma_z$ in that reference) is specified as
$R_\mathrm{s}(\sigma_h) = \begin{pmatrix} 1 & & \\ & 1 & \\ & & -1 \end{pmatrix}, \quad R_\mathrm{o}(\sigma_h) = \begin{pmatrix} -1 & & \\ & -1 & \\ & & 1 \end{pmatrix}$
So $\hat{x} \to \hat{x}$ and $k_x \to -k_x$ gives a total minus sign, and the same for the other term, so $k_x \hat{x} - k_y \hat{y}$ transform with a minus sign under $\sigma_\mathrm{h}$. You can find the action under the other generators like this as well.
The order parameter is the pair operator $\hat{\Delta}_{\mathbf{k},\sigma\sigma'} = \hat{c}_{\mathbf{k},\sigma} \hat{c}_{-\mathbf{k},\sigma'}$, where $\hat{c}$ is the electron annihilation operator. The symmetry group representation acting on $\Delta$ derives from the one acting on $\hat{c}$ as follows. Electrons are fermions, and $\hat{c}$ transforms under a spin or projective representation of the group (say $D_{4\mathrm{h}})$. This representation is not given in most character tables, but for instance the fourth power of the rotation generator is $-1$, not $+1$ in this representation. The representation acting on the order parameter follows from Clebsch-Gordan decomposition of the tensor product of this spin representation with itself, and includes only ordinary (unitary) representations. See the references by Sénéchal below.
Nodes are points in $k$-space where $\Delta(\mathbf{k}) =0$. ($\mathbf{k} =0$ is excluded since pairing is always at some finite momentum, moreover near the Fermi surface.) Therefore, you simply need to inspect where the gap function vanishes.
Looking at the representative functions in Table I of the paper you cite, you can see that $B_{1g}$, $B_{2g}$, $B_{3g}$ vanish at $k_z =0$ for all $k_x$, $k_y$. Therefore, there is a line of nodes at $k_z =0$ as a function of $k_x,k_y$.
Conversely, the $B_{1u}$, $B_{2u}$, $B_{3u}$ representative functions only vanish at some very specific values of $k_x, k_y, k_z$, determined by the coefficients $c_1$, $c_2$. Therefore, these have point nodes. (By the way, there are errors/typos in the representative functions of $B_{1u}$ and $B_{3u}$ in that table. It should be $c_1 k_y \hat{x} + c_2 k_x \hat{y}$ for $B_{1u}$, and similar in $y,z$ for $B_{3u}$.)
The function $z$ transforms under $B_{1u}$ as you mention. But that is not the form of the gap function, which for instance is specified in momentum space. The gap function in question looks like $c_1 k_y \hat{x} + c_2 k_x \hat{y}$, and is independent of $k_z$. If you look in Table V of Kaba and Sénéchal, you can see that there exists a gap function with the form $\hat{b}_z\hat{d}_0 k_z$, also transforming under $B_{1u}$. However $\hat{d}_0$ is the singlet spin-state, and $\hat{b}_z$ is a particular form of the orbital (not spin) part of the order parameter.
References
J. Annett, Unconventional Superconductivity. Contemporary Physics, 36(6), 423 (2005). See especially section 3 for a discussion about the symmetry breaking starting from the full group $G \times SU(2) \times U(1)$ (crystal space group, spin-rotations, and phase-rotations) and a visual representation of the order parameter symmetries, and section 5 for symmetries of the gap function.
S. Sumita. Modern classification theory of superconducting gap nodes. PhD Thesis (2020). Kyoto University. Very thorough treatment. See section 1.1.1 for a short summary of the meaning of the symmetry of the order parameter.
Geilhufe & Balatsky. Phys. Rev. B 97, 024507 (2018). Another nice, concise treatment of transformation properties of the order parameter function, starting from Equation (4). Also includes odd-frequency pairing.
Ishizuka et al. Phys. Rev. Lett. 123, 217001 (2019) See Table 4(a) for the correct basis functions of the irreps of $D_{2\mathrm{h}}$.
S.-O. Kaba and D. Sénéchal. Phys. Rev. B 100, 214507 (2019) See section II.B, III.A for a elaborate derivation of the symmetry transformation properties of the $D_{4\mathrm{h}}$ order parameter, of not only the $d$-vector, but also the orbital part.
D. Sénéchal. Lecture notes A pedagogical treatment of the previous paper.
