I also had the experience of "Why would you define measurements that way?" when learning about Hermitian observables.
At first, I just avoided them. I'd translate observables into a unitary operation followed by a measurement in the computational basis, and think about it that way. For example, for me the Z observable was "just measure" while the X observable was "apply Hadamard, then measure". And the $X \otimes X$ observable was "hit both involved qubits with a Hadamard, CNOT them onto some third qubit, measure that qubit, then undo the Hadamards".
Eventually it started to bother me that my re-description of the measurements as a circuit was often longer. I mean, just look at how many words it took me to describe what I did for $X \otimes X$! And also I started needing the observable's matrix to answer questions like "if I measure A, will it mess up measuring B?". Then I started noticing how useful they were as a thinking tool, and idioms like "Z-value" and "X-parity" started sneaking into my writing... the observables got to me.
1) What does it mean to consider X and Y as observables? Are they not operations that change the current state to a new one?
Consider this: if you reverse the order of a controlled-Z, you still have the same operation. But if you swap the control and the gate in a CNOT, you don't get the same operation:
So there is a sense in which the Z gate is "the same" as an ON-control, and the X gate doesn't share this property. And it comes down to the fact that, when you breakdown what Z does, it does nothing to OFF states but multiplies the amplitude of ON states by -1.
You can define an alternative control that is "the same" as the X gate. In which case you'll find that you care about the distinction between $|+\rangle = |0\rangle+|1\rangle$ and $|-\rangle = |0\rangle-|1\rangle$, instead of the distinction between ON and OFF. And it just so happens that if you break down how the X gate works into its eigenvalues and eigenvectors, that it leaves $|+\rangle$ alone but multiples the amplitude of $|-\rangle$ by -1. (You can play with X-axis and Y-axis controls in Quirk.)
When you generalize this association between "what you leave alone" and "what you affect" to apply to any operation, you end up talking about the eigenvalues and eigenspaces of those operations. And this leads pretty quickly into caring about which eigenspace of an operation a state lies in, and to measuring that information, and then to just thinking of the operation as a specification for the measurement of its eigenspaces.
Physicists happen to care about the logarithm of a unitary operation more than the operation itself, because you can plug it into differential equations. And the logarithm form has other nice properties. So we tend to talk about observables in terms of the logarithm of a unitary matrix, i.e. a Hermitian matrix, instead of directly in terms of the unitary operation.
2) Why does applying X to |0⟩ result in a non-zero standard deviation if X|0⟩=|1⟩? How is there any variation here?
Because you're mixing up the operation X with the observable X.
The operation X toggles between ON and OFF. If you take its eigendecomposition, you find it leaves $|+\rangle$ alone while negating $|-\rangle$.
The observable X is a description of a measurement that distinguishes between the eigenspaces of the operation X. That is to say, it measures whether the system is in the $|+\rangle$ state or in the $|-\rangle$ state.
$|0\rangle$ is neither $|+\rangle$ nor $|-\rangle$, it's a superposition of both, so when you measure its X-value you get variance. States with no X-value variance don't get toggled by X, they get phased.