Suppose that there existed no charge , i.e. up and down quarks were neutral with respect to charge but had the mass they have.
up 2.3 MeV/c^2
down 4.8 MeV/c^2
The neutron is udd , so the mass contribution is 11.9 MeV/c^2 mass of neutron 939.6 Mev?c^2
the proton uud has a mass contribution________9.4 MeV/c^2 the mass of proton 938.3
difference of sum of constituent masses : 1.5MeV/c^2
difference of neutron and proton masses :1.3 MeV/c^2
As far as the strong interactions go it is the color charges that play a role in how many sea quarks and gluons are swirling around generating the almost a hundred times extra mass.
There is 0.2 MeV/c^2 difference in proton and neutron masses. The extra charges in the proton diminish the effect of the sum of the constituent masses, i.e. create 0.2 MeV/c^2 extra mass for the proton.
Let me suppose that this 0.2 MeV/c^2 excess difference between a neutron,which has two down quarks swirling around one up quark, and a proton which has two down quarks swirling around one up, is due to more quark gluon sea energy for the proton because the repulsion of the charges will be more, ( larger charges are involved). Then one could take this 0.2 MeV/c^2 excess of the proton mass with respect to the neutron as the effect of the charges. This would give an upper limit for how smaller the mass of a chargeless proton could be .
It seems reasonable to expect that the existence of charges will introduce more energy into a system's binding energy in the case of the proton, since it is a repulsion of like charges of uu whereas the like charges in the neutron dd are of smaller value. In addition the ballpark of the magnitude, 0.2MeV/c is about 10^-4 of the proton mass, compatible with the 5*10^- 5 coupling-constant-squared of the electromagnetic interaction, which would give the relative strength of the electromagnetic contribution versus the strong force.