"Isoscalar factor" is another name for reduced Clebsch-Gordan coefficients. The nomenclature was often used in the late 60s in connection with SU(3) models, reduced either using SO(3) (as in the Elliott model of nuclear physics) or using SU(2) (as in the familiar gauge theories). In these cases, the isoscalar factors would be multiplied by an SO(3) or SU(2) ClebschGordan coefficient to obtain the full SU(3) Clebsch Gordan coefficient. (Nota: although su(2) and so(3) are isomorphic at the algebra level, at the group level they are distinct: the SO(3) group would be similar to the one found in the angular part of the 3D harmonic oscillator wavefunctions and would be restricted to true representations, i.e. integer values of L, whereas the SU(2) groups can have projective (or spinorial) representations where J is half integer. For SO(3) the embedding inside SU(3) is irreducible but not for SU(2).)
When labelling states using a subgroup chain like SO(n)$\to$SO(n-1), it is also possible to express the Clebsch-Gordan coefficient as a product of an SO(n-1) Clebsch multiplied by an "isoscalar factor", which is the part of the SO(n) Clebsch not in SO(n-1).
In the specific case of SO(3), the subgroup usually used is SO(2): the subgroup of rotations about the z-axis. This subgroup is Abelian so the representations are 1-dimensional, labelled by M, and just exponentials $e^{iM \varphi}$ for instance. For SO(2) the Clebsch are thus 1 if the projections satisfy the correct addition rule $M_1+M_2=M_3$. Thus, in this case, the isoscalar factor for SO(3) is the Clebsch itself since the SO(2) Clebsch is just 1 (or more correctly a delta function enforcing the condition $M_1+M_2=M_3$.)
In the case of SO(n) the situation is complicated because some representations will contain more than one copy of some representations of SO(n-1). The general definition of the isoscalar factor is as above although, if there is more than one copy of an irrep of SO(n-1), an additional label must be used (this also occurs in some irreps of SU(3) when using SO(3) as a subgroup of SU(3), but never occurs when using SU(2) as a subgroup. Again, the physical angular momentum is really SO(3) so the use of this subgroup is sometimes forced upon the user, especially in nuclear physics where 3D harmonic oscillator states are useful). In some specific cases where there is no such multiplicity of representation then of course this additional label is not needed so only the generalized angular momentum labels for SO(n) and SO(n-1) irreps are needed.
