This is a detailed derivation of the final equation for OP's first question.
The spherical harmonics can be viewed as a function, with input $\mathbf{\hat{n}}$ and output $\langle{\mathbf{\hat{n}}}|{lm}\rangle$. The directional eigenvectors are in one-to-one correspondence with their three components in the Cartesian basis $(n_x,n_y,n_z)$, so we may as well use them for the function input:
\begin{align}
Y_l^m: (n_x,n_y,n_z)\to \langle{\mathbf{\hat{n}}}|{lm}\rangle\,.
\end{align}
We can extend the domain of this function by allowing the input to be vector operators, or equivalently the three components of them. The generalized definition is
\begin{align}
Y_l^m: (V_x,V_y,V_z)\to \langle{\mathbf{\hat{n}}}|{lm}\rangle\,,\quad\text{replace $n_{x,y,z}$ by $V_{x,y,z}$}\,.
\end{align}
The extension of $Y_l^m$ is well defined if we specify its expression with noncommutative terms of $n_{x,y,z}$.
For a directional eigenket under rotation,
\begin{align}
\mathbf{\hat{n}}\cong\left[\begin{matrix}
n_x\\n_y\\n_z
\end{matrix}\right]\to \mathbf{\hat{n}}'\cong\left[\begin{matrix}
n'_x\\n'_y\\n'_z
\end{matrix}\right] = R \left[\begin{matrix}
n_x\\n_y\\n_z
\end{matrix}\right]\,.
\end{align}
Here we use $\cong$ to imply the representation under Cartesian basis.
So the function is
\begin{align}
Y_l^m: R \left[\begin{matrix}
n_x\\n_y\\n_z
\end{matrix}\right] \to \langle{\mathbf{\hat{n}}'}|{lm}\rangle\,.
\end{align}
Denote
\begin{align}
\mathbf{V}' = \mathscr{D}^\dagger(R)\mathbf{V}\mathscr{D}(R)\,,
\end{align}
then by definition of vector operator,
\begin{align}
\mathbf{V}'\cong \left[\begin{matrix}
V'_x\\V'_y\\V'_z
\end{matrix}\right] = R \left[\begin{matrix}
V_x\\V_y\\V_z
\end{matrix}\right]\,.
\end{align}
For the input vector operator $\mathbf{V}'$, the function is
\begin{align}
Y_l^m: \left[\begin{matrix}
V'_x\\V'_y\\V'_z
\end{matrix}\right]\to \langle{\mathbf{\hat{n}}}|{lm}\rangle\,,\text{ replace $n_{x,y,z}$ by $V'_{x,y,z}$}\,.
\end{align}
which is equivalently
\begin{align}
Y_l^m: \left[\begin{matrix}
V'_x\\V'_y\\V'_z
\end{matrix}\right] = R \left[\begin{matrix}
V_x\\V_y\\V_z
\end{matrix}\right]\to \langle{\mathbf{\hat{n}}'}|{lm}\rangle = \langle{\mathbf{\hat{n}}}|{\mathscr{D}^\dagger(R)}|{lm}\rangle\,,\text{ replace $n_{x,y,z}$ by $V_{x,y,z}$}\,,
\end{align}
where we used the definition of the directional eigenket.
So we have
\begin{align}
Y_l^m(\mathbf{V}') & = \langle{\mathbf{\hat{n}}}|{\mathscr{D}^\dagger(R)}|{lm}\rangle\quad \text{replace $n_{x,y,z}$ by $V_{x,y,z}$}\\
& = \sum_{m'}\langle{\mathbf{\hat{n}}}|{lm'}\rangle\langle{lm'}|{\mathscr{D}^\dagger(R)}|{lm}\rangle\quad \text{replace $n_{x,y,z}$ by $V_{x,y,z}$}\\
& = \sum_{m'}Y_l^{m'}(\mathbf{V})\langle{lm'}|{\mathscr{D}^\dagger(R)}|{lm}\rangle\\
& = \sum_{m'}Y_l^{m'}(\mathbf{V}){\mathscr{D}^{(l)^*}_{mm'}(R)}\,.
\end{align}
In the final step, use that $Y_l^m(\mathbf{V})$ is a homogeneous polynomial of the components of $\mathbf{V}$, which is proven on P371 of Gordon Baym Lecture notes on quantum mechanics. So we have
\begin{equation}
Y_l^m(\mathbf{V}') = Y_l^m(\mathscr{D}^\dagger(R)\mathbf{V}\mathscr{D}(R)) = \mathscr{D}^\dagger(R) Y_l^m(\mathbf{V}) \mathscr{D}(R)\,.
\end{equation}
It follows the final equation.