So say we have $\hat{O}=|\phi\rangle\langle\psi|$ and a ket vector $|f\rangle$, where
$\langle x|\phi\rangle=\phi(x),\hspace{1cm}\langle x|\psi\rangle=\psi(x),\hspace{1cm}\langle x|f\rangle=f(x)$
If we write the inner product $\hat{O}|f\rangle$ in terms of an integral over $x$, we have
$|\phi\rangle\langle\psi|f\rangle=|\phi\rangle\int\psi(x)^*f(x)dx$
but I don't understand what this means. What does $|\phi\rangle$ become? Is the result a bra, a ket, or a number?
If we write the inner product $\hat O|f⟩$...
As already stated in a comment, that is not an inner product, but an action of the operator $\hat O$ on the ket $|f\rangle$ producing another ket.
What you are missing is that the resolution of identity, also known as closure relation, $\int |x\rangle\langle x|dx = \mathbf{1}$ was used: $$ \hat O|f\rangle=|\phi\rangle\langle\psi|f\rangle=|\phi\rangle\int \langle\psi|x\rangle\langle x|f\rangle dx=|\phi\rangle\int\psi(x)^*f(x)dx. $$
