That large gauge transformations are not true gauge transformations (i.e. yield physically distinct states) is a purely quantum phenomenon due to a choice of quantization procedure that is present in the cases where there are large gauge transformations. Classically, large gauge transformations are always gauge transformations, i.e. trivial on the physical state space. See also this answer by David Bar Moshethis answer by David Bar Moshe.
Essentially, the special status of large gauge transformations arises from the fact that the quantization procedure for a gauge theory only imposes that applying the generators of gauge transformations to physical states must yield zero, and hence the physical states are invariant under gauge transformations generated by them. But, rather by definition, the transformations generated by the generators only yield the gauge transformations connected to the identity (the exponential map of a Lie algebra maps to the connected components of the corresponding group). Therefore, the quantization procedure by design only imposes invariance of the quantum theory under small gauge transformations.
There is no good reason to demand that the quantum theory be invariant under large gauge transformations because it is well-known that the same classical system can have different inequivalent quantizations, and the large gauge transformations simply become the transformations between these inequivalent quantizations, which seems physically reasonable - given a classical theory, its full quantum theory should be the "sum" of all possible quantizations.