Why are there neutrons in the nucleus of an atom?
Protons are positively charged. The electromagnetic force between protons is repulsive even though the residual strong force from the quark content is attractive, the combined potential is not attractive so as to produce a bound state.
Let us take the simple system of two protons with their repulsive potential. The strong nuclear potential added to the electromagnetic repulsion does not give bound energy levels that could accommodate the quantum numbers of the two protons. The addition of the charge-neutral neutron, which,( even though a composite of charged quarks,) has an overall strong force attraction overcomes the repulsive potential and deepens the the total potential so that it can make a potential well to trap in a bound state the three nucleons, He3, an isotope of the alpha nucleus, in appropriate quantum number states. The Pauli exclusion principle also plays a role . See here.
I know that strong force holds protons and neutrons together in nucleus, but how does that effect a neutron in a way that it doesn't decay?
If it were not for the weak interaction the neutron would also be a stable particle, as is the proton. Thus the answer to "why" has to start with the basic decay of the neutron which depends on the quark interactions with the weak force.
The proton is composed out of (u u d) quarks and the neutron out of (u d d). In both the effective mass measured for quarks and in the particle table the up quark has the smallest mass. ( these masses depend on models, but the standard model is very successful in describing elementary particle interactions)
The up has a mass of 2.3 MeV, and the down 4.8. The up is stable as it cannot decay into a lower mass quark. To change an upquark to a down quark , at least 2.5 MeV of energy has to be provided.
The neutron has two down quarks, and they are free to decay to an up quark because it is energetically favorable, and thus it decays to aproton with a lifetime of 900 seconds when free. A proton's single down could also decay , but there is no stable (u u u) lower energy than the proton state within the standard model interactions. Proton decays need new mediating particles and are predicted in various extended models, and have not been detected experimentally up to now.
A nucleus is a many body problem and its quantum mechanical solutions are approximated by the shell model for example, where an effective potential keeps the nucleons, protons and neutrons, in a bound energy level. Thus in a stable nucleus the neutron will be sitting at an energy level 2.5 MeV below the escape threshold and thus will not decay. In an unstable nucleus the effective potential extends above the zero line and there exist energy levels for the neutron to occupy but also a probability since it is energetically possible, to decay. In a simple quantum mechanical model it could tunnel out and decay, as long as the energy level it is bound in is above 0.
See this link for a shell model description on how the proton and neutron see the ( approximate) potential well, strong for neutron, strong and Coulomb for proton.
One should keep in mind that by pion exchange within the nucleus a proton can turn into a neutron and vice verso, in the details of the interactions. Thus a proton sitting in an energy level above the zero energy has a probability to turn into a neutron by pion exchange which can then decay.
As a nucleus is a many body state, quantum numbers also play a large role, and the Pauli exclusion principle. For example the decay will be inhibited if the resulting proton from the neutron decay has no allowed energy level to occupy.
So the answer to " but how does that effect a neutron in a way that it doesn't decay?" is that it is all dependent on the effective many body potential . Neutrons in energy levels below escape threshold, bound, will not decay, in nuclei where there are energy levels above threshold they have a probability of decay since it is energetically possible, if the quantum number conservations allow it..