Why is there a factor of $\hbar$ in the physicists' $\mathfrak{su}(2)$ Lie algebra? The standard (mathematicians') basis for the $\mathfrak{su}(2)$ Lie algebra is:
$$X_j = -i\frac{\sigma_j}2$$
where $\sigma_j$ are the Pauli matrices. In physics, observables correspond to real numbers, so following the standard procedure we convert these to self-adjoint operators, which have real eigenvalues:
$$X_j \rightarrow iX_j$$
This is straightforward so far. But the convention used is $X_j \rightarrow i\color{red}\hbar X_j$ - although there seems to be no a priori reason to add this factor here. One reference said something along the lines of "The factor of $\hbar$ is an indicator that we're using quantum calculations", but I find this to be a bit vague.
What's the real reason behind this?
 A: I think the physical context of your question is
spin angular momentum of elementary particles (for example: an electron)
and rotations of this particle in space.
Let's begin with spin angular momentum $\bf{S}$ (a vector with 3 components
$S_1$, $S_2$, $S_3$). Physical experiments showed that the measured spin
component $S_j$ of the particle is either $+\frac{1}{2}\hbar$ or
$-\frac{1}{2}\hbar$.
You see, Planck's constant $\hbar$ here (and its extremely small
numerical value $1.055 \cdot 10^{-34}\rm{\ kg\ m^2/s} $) comes from physical experiments.
Hence the observable $S_j$ is represented by the self-adjoint operator
$$S_j=\frac{1}{2}\hbar\sigma_j.$$
The two eigenvectors of this operator are the measured values.
On the other hand you have rotations of the particle in space. These rotations
are represented by unitary elements of the Lie group $SU(2)$.
$$U = e^{\frac{i}{2}(\theta_1\sigma_1+\theta_2\sigma_2+\theta_3\sigma_3)}$$
where $\theta_1$, $\theta_2$, $\theta_3$ are arbitrary real parameters
encoding the axis and angle of the rotation.
These Lie group elements are obviously generated by the 3 self-adjoint generators
$$X_j=\frac{1}{2}\sigma_j$$
which spawn the Lie algebra $\mathfrak{su}(2)$. The definition is chosen without
the $i$ so that the $X_j$ will be self-adjoint.
You see, the observable $S_j$ and the generator $X_j$ are nearly the same,
only different by the factor $\hbar$: $S_j=\hbar X_j$.
By the way: This is essentially the contents of Noether's theorem,
saying every symmetry transformation (here generated by $X_j$)
corresponds to a conserved physical observable (here $S_j$).
Now we physicists are lazy people. And therefore we use $S_j$ for both
purposes ($S_j$ as the spin observable and $S_j/\hbar$ as the rotation generator).
Thus we can write
$$U = e^{i(\theta_1 S_1+\theta_2 S_2+\theta_3 S_3)/\hbar}$$
A: $\hbar=\frac{h}{2\pi}$ is proportional to the Planck constant $h$ and has the dimension of angular momentum, so the observables consist of Pauli matrices multiplied by a $\hbar$ correspond to the angular momentum, which is not just real numbers, but have direct physical meaning. 
Sometimes physicists would ignore the dimension of an obervable and only concerned about the algbra, then they just take $\hbar=1$ or similarly the elementary charge $e=1$ for simplicity, but at the final step the dimension must be recovered to get the value of observable.
A: the reason is that the real world has dimensions. Spin in physics originates from the irreducible representations of the Lorentz group, which is $SO(3,1)$. The Loretnz group relates to transformations of coordinate systems. These transformations correspond to real space-time quantities, and therefore come with dimensions attached to them. Rotations come with dimensions of angular momentum, which is the same dimensions of $\hbar$. So naturally the physical quantity that correspond to rotations, here the spin, should have the same dimensions.
It turns out that the universe has scales in it, and phenomena behave differently when examined at different scales. For example - physics at low velocities is different than at high velocities. When we say "high" velocity or "low" velocity we ask ourselves "with respect to what?" and the answer the universe give us is the speed of light $c$. So we measure velocities relative to that quantity, if th velocity is much smaller than $c$, the physics is classical (can be approximated by Newton's mechanics) and if it comparable to $c$, it is relativistic. Similarly, in angular momentum we measure large and small with respect to $\hbar$. If things are of the scale of $\hbar$ they are quantum, and if they are much larger than $\hbar$ they are classical (that is - can be approximated by classical mechanics).
