I know that even if we have:
$$a |\alpha \rangle = \alpha |\alpha\rangle$$
We don't have:
$$a^{\dagger} |\alpha \rangle = \alpha^* |\alpha\rangle$$
Actually as explained in the second answer here Eigenvalue for the creation operator for a coherent state $a^{\dagger}$ doesn't have eigenvalues.
But what disturbs me is the "matrix" vision.
If I choose a basis composed of $|\alpha\rangle$ and then some other states such that I have an orthogonal basis in the end, I can write down the (infinite) matrix $a$ in this basis.
The first diagonal coefficient will be $\alpha$. Then to go to $a^{\dagger}$ I conjugate transpose the matrix which will give me $\alpha^*$ on the first diagonal element.
Then through this vision it seems that $|\alpha\rangle$ is eigenstate of $a^{\dagger}$ with $\alpha^*$ as eigenvalue.
Where is the mistake in this reasoning ? Is it because we work in infinite dimension space so the matrix vision can induce errors ?