What do we mean by saying product of operators is a scalar? The motivation for this question is considering the helicity of a massless particle in relativistic QFT.
As the definition for helicity operator $h$
$$
h=\frac{\mathbf{J} \cdot \mathbf{P}}{|\mathrm{P}|}
$$
where $J$ and $P$ are three dimensional angular momentum operators and momentum operators respectively.
Now every textbook says that this is a "scalar" under Lorentz transformations for massless particles, which follows from the definition of rotations and boosts.
Now
1)Firstly I just want to be more precise to ask that in what sense do we mean by "scalar" for calling this quantity, since it is a operator rather than a number. Does it mean that the $h$ operator does not change its form under Lorentz transformation?
2)Secondly it is easy to prove that $h$ is invariant under rotations, but how do we prove that it is invariant under boosts? Or do they?
In fact in another question there is an answer saying that the boosts generators do not commute with h in fact

$$
\left[\frac{J \cdot P}{H}, K_{i}\right]=i\left(\frac{\epsilon_{i j k} K_{j} P_{k}}{H}+J_{i}-\frac{P_{i}}{H^{2}} J \cdot P\right)
$$

But on the other hand, massless particles always travel at the speed of light, so my guess is that boosts do not have influence on the massless particles but only rotations do, is this argument the reason why helicity is an invariant quantity for massless particles? And is there any more solid or inspiring explanation on this thing.
 A: As you correctly point out, when saying that a quantity is a scalar, one should also specify respect which group is a scalar. A quantity is a scalar under a certain group action if transformes under the trivial representation of the group (i.e. It is leaved invariant by the transformation). The important aspect is that a scalar is not specifically a "number", energy $E$ for example isn't invariant under a Lorentz boost, while a four vector $v=v^ie_i$, being a good geometrical object, is invariant under Lorentz transformation, hence It is a scalar under $SO(1,3)$
Intuitive argument about helicity invariance under boost for masless particles:
helicity is an invariant quantity under Lorentz boost only for particles without mass. These have speed $c$, hence you can imagine such a particle travelling with a certain value of helicity, given by the projection of spin along the direction they are moving. Being $c$ its velocity, you can't imagine an external observer boosted in such a way that surpass the particle. If this could happen the observer would see the particle change the direction $p \rightarrow -p$ and so there would be a helicity flip, and $h$ wouldn't be invariant. But that is impossible because the particle has null mass while the observer doesn't have null mass.
Edit:
here is a more complete argument, just to try to give you a direction to look, (if I can I'll try to edit the answer to explain this more completely).
In the massless limit helicity reduces to chirality. These two quantities are equivalent only for $m=0$, and in the general case of $m \not = 0$ helicity isn't lorentz invariant, while chirality is. The operator $h$ you wrote is still the same operator that gives the helicity eigenvalue of massive eigenstate. So you are trying to prove helicity invariance under boost, for masless particles, without changing the form of the operator $h$ respect this limit. In the limit of null mass this operator $h$ reduce to the chirality operator $\gamma^5$ which instead is a lorentz invariant quantity, as said before. I hope this can helps you
