Rob's answer is the simplest and probably best, but let me add another approach.
We know that nuclei are made out of protons and neutrons. Protons repulse each other, but somehow, if you get them close enough, they stick together extremely strongly. This already suggests that there is another force in play! So even if you completely ignored neutrons, you would need some strong force that overcomes electromagnetism at sufficiently small distances. Of course, the simplest multi-proton nucleus, the diproton, is relatively unstable - but it's still stable enough to allow our Sun to work; it lasts long enough for one of the protons to very rarely change into a neutron, forming the stable deuterium. Interestingly, if the strong force was a tiny bit stronger, diproton (He-2) would be stable.
Now, neutrons are relatively easy to experiment with - you can shoot free neutrons at targets and see what happens. If you shoot an electron through a cloud chamber, it will leave a trail throghout its path (this is one of the main ways we observe very small amounts of stuff). If you add an electric field, the electron's path will be deflected - it will be attracted or repulsed from the source of the field (e.g. a magnet). The neutron isn't.
But that's what Rob already said, so let's assume we can't observe free neutrons this way. Would the neutron's behaviour be consistent with electromagnetism?
Neutrons mostly don't affect chemistry. But if they balanced out the electromagnetic charge from the protons, the number of electrons in an atom would depend on both the number of protons and neutrons (more neutrons would mean less electrons). You wouldn't have isotopes (or rather, it would mean something different). So to get this to work, you'd somehow have to have the electrons to ignore the charge of the neutron. This already means you need to have another force, one that doesn't affect electrons.
But let's keep on going despite this impossibility. If neutrons had a strong negative electromagnetic charge (that somehow ignored electrons), they would be very strongly attracted by atomic nuclei. This is not what we actually observe - you need to hit the neutron essentially head on to the nucleus to get it absorbed (the target area is called the neutron cross section). Free neutrons wouldn't penetrate much into matter, since even very fast moving neutrons would be rapidly deflected or absorbed by nuclei.
Both Helium-3 and Helium-4 are stable. But they have the same number of protons, while one has a single neutron, and the other has two. But if neutrons attract protons through the electromagnetic force, they must also repel each other. Regardless of what charge you set for the neutron, it would need to be able to balance out the charge of the two protons trying to repel each other, but at the same time, adding another neutron would not cause the nucleus to fall apart. And remember that we're still ignoring the electrons - somehow, despite the protons not repulsing each other thanks to the neutron's negative charge, the electrons are still atracted and bound to the nucleus.
Finally, if neutrons actually worked the way you posit, we would already have nuclear fusion! It would be even easier than nuclear fission. Indeed, in your scenario, nuclear fission would be almost impossible, while fusion would be trivial. Dropping a free neutron into a target atom would release huge amounts of energy. In fact, nuclei would clump together spontaneously even at standard conditions - a hydrogen nucleus would be repelled from another hydrogen nucleus, but as soon as you add a neutron into the mix, its attraction would necessarily be much larger than the repulsion between the protons. A deuteron would spontaneously combine with a free hydrogen nucleus, at room temperature and pressure, releasing vast amounts of energy.
If we magically replaced the strong force with an electromagnetic charge on the neutron (while keeping the binding energy in existing nuclei the same), all the matter around you would collapse into one massive nucleus at velocities close to the speed of light, while releasing humongous amounts of energy, before everything collapses into massive black holes.
To explain the observations, you need a force that is very strong between protons and neutrons at very short distances, but doesn't affect electrons and very quickly drops off after some critical distance. We call that force the strong nuclear force. This is the simplest explanation that fits all the observed data (and has proven itself over and over with advanced predictions that have been confirmed by experiment), so it "wins".
Interestingly, the so-called weak force is thought to be even stronger than the strong force - but only on even shorter distances. Both of these strange behaviours have been critical in expanding our understanding of how the universe works.