Given that $O_{ij}$ is the entry of the matrix $O$ at the $i$-th row and $j$-th column, the operator $\hat{O}$, from which you obtain the matrix $O$ by projecting it on elements $|i\rangle$ and $|j\rangle$ out of an orthonormal basis spanning the Hilbert space on which $\hat{O}$ acts, can be written as

$$\hat{O}=\sum_{ij}\langle i |\hat{O}|j\rangle |i\rangle \langle j|==\sum_{ij}O_{ij} |i\rangle \langle j|.$$

So, in your case $$\hat{U}_{\alpha}=\cos(\alpha)|0\rangle \langle 0|+ i\sin(\alpha)|0\rangle \langle 1|+i\sin(\alpha)|1\rangle \langle 0|+ \cos(\alpha)|1\rangle \langle 1|$$ acts on $|0\rangle$ as $$\hat{U}_{\alpha}|0\rangle=\cos(\alpha)|0\rangle \langle 0|0\rangle+ i\sin(\alpha)|0\rangle \langle 1|0\rangle +i\sin(\alpha)|1\rangle \langle 0|0\rangle + \cos(\alpha)|1\rangle \langle 1|0\rangle.$$ Since $\langle 0|0\rangle=1$ and $\langle 1|0\rangle=0$, $$\hat{U}_{\alpha}|0\rangle=\cos(\alpha)|0\rangle + i\sin(\alpha)|1\rangle.$$

Given that $O_{ij}$ is the entry of the matrix $O$ at the $i$-th row and $j$-th column, the operator $\hat{O}$, from which you obtain the matrix $O$ by projecting it on elements $|i\rangle$ and $|j\rangle$ out of an orthonormal basis spanning the Hilbert space on which $\hat{O}$ acts, can be written as

$$\hat{O}=\sum_{ij}\langle i |\hat{O}|j\rangle |i\rangle \langle j|=\sum_{ij}O_{ij} |i\rangle \langle j|.$$

So, in your case $$\hat{U}_{\alpha}=\cos(\alpha)|0\rangle \langle 0|+ i\sin(\alpha)|0\rangle \langle 1|+i\sin(\alpha)|1\rangle \langle 0|+ \cos(\alpha)|1\rangle \langle 1|$$ acts on $|0\rangle$ as $$\hat{U}_{\alpha}|0\rangle=\cos(\alpha)|0\rangle \langle 0|0\rangle+ i\sin(\alpha)|0\rangle \langle 1|0\rangle +i\sin(\alpha)|1\rangle \langle 0|0\rangle + \cos(\alpha)|1\rangle \langle 1|0\rangle.$$ Since $\langle 0|0\rangle=1$ and $\langle 1|0\rangle=0$, $$\hat{U}_{\alpha}|0\rangle=\cos(\alpha)|0\rangle + i\sin(\alpha)|1\rangle.$$

Given that $O_{ij}$ is the entry of the matrix $O$ at the $i$-th row and $j$-th column, the operator $\hat{O}$, from which you obtain the matrix $O$ by projecting it on elements $|i\rangle$ and $|j\rangle$ out of an orthonormal basis spanning the Hilbert space on which $\hat{O}$ acts, can be written as

$\hat{O}=\sum_{ij}\langle i |\hat{O}|j\rangle |i\rangle \langle j|==\sum_{ij}O_{ij} |i\rangle \langle j|$.

So, in your case $\hat{U}_{\alpha}=cos(\alpha)|0\rangle \langle 0|+ isin(\alpha)|0\rangle \langle 1|+isin(\alpha)|1\rangle \langle 0|+ cos(\alpha)|1\rangle \langle 1|$ acts on $|0\rangle$ as $\hat{U}_{\alpha}|0\rangle=cos(\alpha)|0\rangle \langle 0|0\rangle+ isin(\alpha)|0\rangle \langle 1|0\rangle +isin(\alpha)|1\rangle \langle 0|0\rangle + cos(\alpha)|1\rangle \langle 1|0\rangle$. Since $\langle 0|0\rangle=1$ and $\langle 1|0\rangle=0$, $\hat{U}_{\alpha}|0\rangle=cos(\alpha)|0\rangle + isin(\alpha)|1\rangle$.

Given that $O_{ij}$ is the entry of the matrix $O$ at the $i$-th row and $j$-th column, the operator $\hat{O}$, from which you obtain the matrix $O$ by projecting it on elements $|i\rangle$ and $|j\rangle$ out of an orthonormal basis spanning the Hilbert space on which $\hat{O}$ acts, can be written as

$$\hat{O}=\sum_{ij}\langle i |\hat{O}|j\rangle |i\rangle \langle j|==\sum_{ij}O_{ij} |i\rangle \langle j|.$$

So, in your case $$\hat{U}_{\alpha}=\cos(\alpha)|0\rangle \langle 0|+ i\sin(\alpha)|0\rangle \langle 1|+i\sin(\alpha)|1\rangle \langle 0|+ \cos(\alpha)|1\rangle \langle 1|$$ acts on $|0\rangle$ as $$\hat{U}_{\alpha}|0\rangle=\cos(\alpha)|0\rangle \langle 0|0\rangle+ i\sin(\alpha)|0\rangle \langle 1|0\rangle +i\sin(\alpha)|1\rangle \langle 0|0\rangle + \cos(\alpha)|1\rangle \langle 1|0\rangle.$$ Since $\langle 0|0\rangle=1$ and $\langle 1|0\rangle=0$, $$\hat{U}_{\alpha}|0\rangle=\cos(\alpha)|0\rangle + i\sin(\alpha)|1\rangle.$$