I think the main issue is that you are being confused by the Dirac notation, which works fine when it does but can be occasionally confusing when you are worried about that kind of thing. I'll try and give an accessible explanation that also addresses the issues raised by Wouter.
Take a Hilbert space $\mathcal{H}$, and a linear operator $\hat{A}:\mathcal{H}\rightarrow\mathcal{H}$. The Hilbert space comes equipped with an inner product $\langle\cdot,\cdot\rangle:\mathcal{H}\times\mathcal{H}\rightarrow\mathbb{C}$, and this inner product associates to each vector $\psi\in\mathcal{H}$ a linear functional $\langle\psi,\cdot\rangle:\mathcal{H}\rightarrow\mathbb{C}$, which acts in the natural way. Thus there is a correspondence between the Hilbert space and the set $\mathcal{H}^\ast$ of all linear functionals from $\mathcal{H}$ into $\mathbb{C}$. This space is known as the dual space of $\mathcal{H}$ and is where (well-behaved$^1$) kets live.
Consider then a fixed vector $\phi\in\mathcal{H}$ and a linear operator $\hat{A}: \mathcal{H} \rightarrow\mathcal{H}$. These two define a new linear functional which depends only on the one associated with $\phi$:
$$
\psi\mapsto\langle\phi,\hat{A}\psi\rangle.
$$
If you denote by $\phi^\dagger=\langle\phi,\cdot\rangle\in\mathcal{H}^\ast$ the linear functional associated with $\phi$, then the new functional defined above is a function of $\phi^\dagger$: it is denoted $\hat{A}^\dagger\phi^\dagger\in\mathcal{H}^\ast$, and this function $\hat{A}^\dagger$ is called the adjoint of $\hat{A}$.
Now, since the Hilbert space and its adjoint are in a strict correspondence, we can pull this back to define the action of the adjoint on $\mathcal{H}$ itself. This is done in the obvious way: $\hat{A}^\dagger$ acting on an arbitrary vector $\phi\in\mathcal{H}$ gives the unique vector $\hat{A}^\dagger\phi\in\mathcal{H}$ such that for any $\psi\in\mathcal{H}$ you have
$$
\langle\hat{A}^\dagger\phi,\psi\rangle=\langle\phi,\hat{A}\psi\rangle.
$$
This is really the definition of the adjoint $\hat{A}^\dagger$.
Since the adjoint $\hat{A}^\dagger$ acts on the same space as the original operator $\hat{A}$, they are comparable and we can ask whether they are equal. With the definition just given, the condition for that is
$$
\hat{A}\textrm{ is hermitian} \Leftrightarrow \hat{A}=\hat{A}^\dagger\Leftrightarrow \langle\hat{A}\phi,\psi\rangle=\langle\phi,\hat{A}\psi\rangle\textrm{ for all }\phi,\psi\in\mathcal{H}.
$$
The argument I gave above works only for the left-to-right sense of this equivalence. To prove the converse you need the theorem @MarkMitchison mentioned (in essence) earlier, which states modulo subtleties that two operators are equal if and only if all their matrix elements are equal. That is:
$$
\hat{A}=\hat{B}\Leftrightarrow \langle\phi,\hat{A}\psi\rangle=\langle\phi,\hat{B}\psi\rangle\textrm{ for all }\phi,\psi\in\mathcal{H}.
$$
That's as far as the maths is concerned; now for some physics. Why did I state Dirac notation can be confusing? Well, when you come across a matrix element like $
\langle\phi|\hat{A}|\psi\rangle$, it can be hard to see what exactly is acting on what. In the terms laid out above,
$$
\langle\phi|\hat{A}|\psi\rangle
=\langle\phi,\hat{A}\psi\rangle
=\langle\hat{A}^\dagger\phi,\psi\rangle
=\left(\hat{A}^\dagger\phi^\dagger\right)(\psi).
$$
The last term means the linear functional $\hat{A}^\dagger\phi^\dagger$ acting on the vector $\psi$. This is what is meant by saying that $\hat{A}^\dagger$ acts on bras.
$^1$I'm going to ignore the fact that functionals may be discontinuous, operators may be unbounded, domains may be restricted, symmetric operators need not be hermitian, and such other difficulties. All of this can be rigorously dealt with using rigged Hilbert spaces. For resources on that see e.g. this question.