As I understand there are eight types of gluons (linear combinations of color/anticolor pairs with varying amplitudes) which can combine (for very short periods) to form glueballs. If there were no restrictions on which types of gluons could be combined, there would be 36 types of glueballs with 2 gluons, 120 types with 3 gluons, 330 types with 4 gluons, etc. But glueballs must be color neutral, so not all of these types are possible.
How many glueballs are possible with a given number of constituent gluons? I think this is really two questions, since it's possible that a glueball A+B and a glueball C+D would be indistinguishable despite {A, B} not being equal to {C, D}, and so there could be distinguishable vs. indistinguishable counts.
If it helps I have a graduate understanding of algebra (though mostly field theory, not group theory) albeit essentially no knowledge of physics. Since there is an explicit representation for SU(3) it may suffice to explain how to determine whether a given combination of matrices is color neutral. Maybe there are better ways.
A related question, Permissible combinations of colour states for gluons, asks where the 8 comes from; mine is essentially asking the next step after that.