For more understanding, you should learn a bit of Sturm-Liouville theory of differential equations to understand why we care about such things.
What is the advantage of having a hermitian operator in QM? What is the most useful part of having a hermitian operator?
When initially trying to build a new theory, it is best to be artificially excessively restrictive so as to be guaranteed good results. It is known before von Neumann formulated modern quantum theory in the current mathematical formalism, that Hermitian operators always have
- complete set of eigenfunctions suitable to expand the entire Hilbert space
- real valued eigenvalues so that all measurements immediately make physical sense
It turns out that we do not need to expand beyond Hermitian operators to work out all the consequences of quantum theory, and so we do not. Note that we actually do have known situations whereby entertaining non-Hermitian quantum mechanics is more convenient, but that is a niche thing, and we do not have to entertain the philosophical headaches that that would bring to us.
How does an operator act on a $\left|\psi\right>$ in terms of measurement of the system?
The result of a measurement is literally the content of a postulate.
So does $\left|\psi(\vec p)\right|^2$ tells us about the probability of having the momentum $\vec p$ at time of measurement? If true, can we generalize it to other operators as well?
Yes and yes.
For example, there is a normalized wavefunction $\psi(x)$, does each $x$ describes the state of particle? [...] If not, then what is meant by "state" in Quantum mechanics?
No. $x$ is the position coördinate of spacetime. I suppose you would like to consider infinitely localised states $\psi_{x_0}(x)=\delta(x-x_0)$ as a state, in which case you would have your infinite set of states to expand over, one for every $x_0$. However, this is not what is being meant.
Instead, the entire $\psi(x)$ itself is the (pure) state of quantum particle. Now, to be even more pedantic, $\psi(x)=\left<x|\psi\right>$, and it is $\left|\psi\right>$ that is the (pure) state of the quantum particle, whereas we pretend that $\left<x\right|$ to be a suitable position coördinate basis expansion for that state.
Note that a state would define a probability amplitude distribution over the entirety of space at once.
what does $\left|\psi^\prime(x)\right|^2$ represents?
Note that when $\psi(x)$ is normalised properly, then the derivative is not normalised. So this would not be a probability; at most it is a weighted, scaled, proportional thing to the probability. With a bit of thought, it is the position probability distribution multiplied by square-momentum so that it gives the weight of how much momentum squared is at which position in space. If you integrate this over all positions, then you get the expectation square-momentum value of the state. It is of use in writing down the kinetic energy of a state, in the non-relativistic approximation.