This is a question from Irodov's Problems in General Physics.
The Rydberg formula for calculating the wave number of a spectral line is: $$ n = R(1/n_f^2-i/n_i^2)\,. $$ where $n_f$ and $n_i$ represent the final and initial states of the electron respectively, and $R$ is the Rydberg constant.
So for a given value of $n_f$ and a given value of $n_i$, you get one particular spectral line. Change $n_i$ or $n_f$ (or both) and you get a different spectral line.
In the above problem, $n_f$ and $n_i$ can have any integral value from 1 to $n$. The number of spectral lines is simply the number of ways in which you can "choose" two distinct integers from 1 to $n$. That would be $^nC_2$.
However, what if two spectral lines have exactly the same wave number? In other words, what if $$ 1/n_1^2-i/n_2^2=1/n_3^2-i/n_4^2\, $$ where $n_1$, $n_2$, $n_3$, $n_4$ are distinct integers from 1 to n? In this situation, the two spectral lines would "overlap", and it makes no sense to count them twice when they are actually just the same line. So, shouldn't the actual answer be less than $^nC_2$?