In particular, you know that $x$ and $p$ are self adjoint, meaning that $x = x^{\dagger}$ and $p = p^{\dagger}$.
If you wanted to calculate the adjoint of an operator like $a$, you would need to complex-conjugate and transpose everything in it. You know how to do that for $x$ and $p$, so what about the $-i$ in there? It gets complex-conjugated only (you can't transpose it because it's a number!).
So $a^{\dagger} = x^{\dagger} + ip^{\dagger} = x + ip$. Note that $a$ is not self-adjoint, because $a \neq a^{\dagger}$.
Is there any general mechanism to find the adjoint of an operator without using any basis?
If $A$ is an operator in a Hilbert space, $A^{\dagger}$ makes sense even when you are not talking about a matrix explicitly, because the $\dagger$ operation is something you perform on operators in general. Matrices are just representations of operators in a certain basis so the $\dagger$ operation takes a certain form to operate on them (as noted, it is complex-conjugating and then transposing).
For operator objects in general, $A^{\dagger}$ means that the operator must act to the left on a bra like $⟨ψ|A^†$ (as opposed to $A$ that must act to the right on kets like $A|ψ⟩$).
To know what form $A^{\dagger}$ will take explicitly you need to know what effect it has on a set of chosen states, meaning you need to choose a basis.