Generalised Coordinates in 3D Rotation If you have N particles on a surface of a rigid body and the rigid body is rotating about some axis, we say there are six generalised coordinates for the system (N particles on the surface) and set up the lagrangian.
The constraints we know are 


*

*The distance between any two particles is invariant 

*The angles between the line joining any particles is invariant as well
and that's it.
We also know a formula for finding the number of genralised coordinates 
i.e difference between number of degrees of freedom (3N) and number of constraints (which is N(N-1)/2).
Clearly using this formula doesn't give 6 generalised coordinates.
Where's the mistake and how to count the number of generalised coordinates?
Note : Even I know the argument as to if you know 3 points on surface, you can determine the position of any other particle on the surface. But my question is about counting the generalised coordinates using the above formula.
 A: Your mistake is to assume that the number of constraints is $N(N-1)/2$. This number is only true for $N\leq 4$. 
Let us label the particles by $1,2,\ldots,N$ and say that $\bar{ij}$ denotes the constraint between particles $i$ and $j$. For one particle there is no constraint. For two particles there is one constraint (see the diagrams bellow). If $N=3$ we have three constraints: $\bar{12},\bar{13},\bar{23}$. Add a fourth particle and we gain three others constraints: $\bar{14}$ and $\bar{24}$ fix the distance from particle $4$ to particles $1$ and $2$ and $\bar{34}$ which forbids the fourth particle to rotate about the axis joining $1$ and $2$. This fixes the distance from $4$ to $3$. This gives a total of six constraints. Finally consider adding one more particle. Now things get different. The new constraints $\bar{15},\bar{25}$ fix the distance to $1$ and $2$. The only freedom the sixth particle still have is to rotate about the line joining $1$ and $2$. Therefore just one more constraint, say $\bar{13}$, is enough to rigidly fix the sixth particle. The total number of constraints in this case is six.

As you can see in this construction, the first particle added zero constraints, the second one added one constraint, the third one added two constraints and the nth ($n\geq 4$) one added three constraints. The total number of constraints for $N\geq 5$ particles is therefore $1+2+3(N-3)=3(N-2)$. Hence there are $3N-3(N-2)=6$ degrees of freedom.
