The quarks have fractional charge not only because of fitting proton and neutron constituents. There is a huge data base of hadronic resonances that gave rise to to the quark model , the standard SU(3)xSU(2)x(U(1) of particle physics.
The quantum numbers assigned to the quarks are important, among them the 1/3 and 2/3 charge and the color assignements. The representations of the resonances would not fit the model with different charges.
An example of the symmetries that led to the quark model"
The S = 3⁄2 baryon decuplet
The Omega minus was one of the first predictions of the quark model, before it was found in Brookhaven bubble chamber data.
Edit after comments.
Why is the above decouplet structure ( and all the other representations in the link ) "evidence" for the fractional charge of quarks?
These structures are described by a group called SU(3) . This has a rigid representational structure, more complicated than a crystal which has simple symmetries. The fact that all the known hadronic resonances have a niche in one of these representations, allows to identify the constituents of the hadrons with the basic unit vectors of the SU(3) symmetry, as three "quarks".
The resonances in their niches have charges. In order for the charges to be consistent the quarks have to be identified with fractional charge , to add up to 1 and -1 and 0 ( integer numbers), and in addition, the complexity of the representations (each niche is accompanied by group constants) imposes the fractions to be either +/- 1/3 or +/-2/3 in order to fit the resonances in the appropriate niches with the correct charge. Otherwise the symmetry would break, the puzzle pieces would not fit.
The fact that the Omega minus was predicted and then was discovered, clinched the model, although the beautiful symmetries had already convinced most physicists at the time.