Can a Hermitian operator $A$ bring a wavefunction $|\psi \rangle$ out of Hilbert space? I recently learned about an example where an operator $O$ (not Hermitian) acted on a wavefunction $|\psi \rangle$ and carried the wavefunction out of Hilbert space, i.e. $O|\psi \rangle$ is not in Hilbert space.
Can a Hermitian operator $A$ carry a wavefunction out of Hibert space? 
Up till now, I always did calculations in QM assuming that $A|\psi \rangle$ is still in Hilbert space. 
 A: The answer is negative from scratch. An operator $A:D(A) \to H$ with $D(A)$ a linear subspace of $H$ is said Hermitian when $\langle Ax|y\rangle= \langle x| Ay\rangle$ if $x\in D(A)$. It is clear that $Ax \in H$ in any cases if $x$ is an element of the Hilbert space such that $Ax$ is defined. Also the Hermiticity  requirement is quite irrelevant in this discussion. 
This sort of issues just arises from some confusion about basic mathematical tools used in physics. I admit that sometimes, or also often, physical motivations play a crucual propulsive  role  in developing the formalism, but once the basic formalism is established, similar issues  (discussed as they deserve) should be eventually  viewed as nonsensical. 
There are however  many cases where apparently operators bring their  argument outside the Hilbert space as clearly illustrated in the other answer by Mike Stone which I interpret as a very good practical   discussion about the  reason why this apparent paradox pops  out in the standard manipulation of the mathematical formalism of physicists.
(It is however false that $p^4 \neq p^2 p^2$ when properly using standard  mathematical definitions regarding compositions of operators).
However in a strict sense, referring to the mathematical formalism of Newtonian physics, an issue similar to that raised here could be "is there a triangle with four edges?". 
A: Formally self-adjoint, but  unbounded, operators can easily take a normalizable state (i.e a state in the Hilbert space) and make it unnormalizable and therefore no longer in the space. This can lead to all sorts of aparent paradoxes. For example consider the operator $p^4= \partial_x^4$ on infinite square well  $[0,1]$. Let it act on the  wavefunction 
$$
\psi(x)=\sqrt{30} x(1-x)= \sum_{n=0,{\rm odd}}^\infty \frac{\sqrt{960}}{n^3\pi^3} \phi_n(x)
$$
where $\phi_n(x)= \sqrt{2}\sin n\pi x$ are the normalized eigenfunctions of $p^2=-\partial_x^2$. 
This wavefunction statisfies the usual boundary condition of a "particle in a box," and is normalized. 
Clearly differentiaing $\psi(x)$ four times gives zero, but acting four times on the eigenfunction expansion gives 
$$
p^4 \psi=  \sum_{n=0}^\infty \sqrt{960}n\pi  \phi_n(x)
$$
which has $||\psi||^2 \propto \sum_{n=1}^\infty n^2 = \infty$, so the resultant state is no longer in the Hilbert space.
For this reason, and others, unbounded operators are not allowed to act on all states in the Hilbert space, but instead have a domain which is at best a dense linear subspace of the Hilbert space, and always includes the restriction that the action of the operator on any state remains in the Hilbert space. In the example above $\psi$ is in the domain of $p^2$ as $p^2\psi$ is normalizable, but it is not in the domain of $p^4$.  Therefore $p^4\ne (p^2)^2$ because the output of $p^2$ no longer in the domain of the second $p^2$ factor. 
The notion of a  "Hermitian" operator, although popular in intro QM courses, is  unsafe as "Hermitian",   only impies that $p$ and $p^\dagger$ are given by the same differential operator.The better notion is  self-adjoint because it requires that $p$ and $p^\dagger$ are not only given by then same formuala, but also that they also have the same domain. Any quantum obsservable has to be self-adjoint. There are many operators that are hermitian, but do not have a complete set of eigenfunctions, and therefore have no meaning in  Quantum mechanics.
