You're double counting here. Lets take three particles. You're counting $\binom{3}{2}=3$ DOFs, right? But fixing the vector distance between particle 1 and two, and then fixing it between 2 and 3 includes fixing it between 1 and 3. Mathematically, $\vec{d}_{1,3}=\vec{d}_{1,2}+\vec{d}_{2,3}$
The easier way to count DOFs is like this. For a molecule with N particles, number of DOFs is $3N$. Out of these, 3 will be translational. For a point molecule (i.e, a single atom), subtract 3 as it has 0 rotational DOFs. For a perfectly linear molecule, subtract 1, as it has 2 rotational DOFs (Rotation along its axis is irrelevant). Now, we usually neglect vibrational DOFs (at normal temperatures). Vibrational DOFs are whatever DOFs are remaining. Thus, we always have a total of 3N DOFs, out of which we may count only the translational (3) and rotational (2 or 3) DOFs. See the table here.