It's important to distinguish between two very different regimes when considering the stress-strain behavior of metals: (1) The elastic regime and (2) the plastically deforming regime. When relatively small stresses are applied to metals, they tend to behave elastically. If you apply a stress it bends a little, and if you then remove the stress it goes back to its original position. Elastic properties of metals depend on the elemental composition of the metal, but they tend to be insensitive to the microstructural details of the metal. Things like dislocations (produced by work hardening a metal) or fine precipitates (which can be produced by age hardening the metal) don't affect the elastic properties of metals like the Young's modulus much since they tend to be a relatively small volume fraction of the overall volume of the metal.
So the question is really why do these microstructural details become so important when the metal starts to plastically strain? When the stress on a metal becomes large enough to plastically stain it, we enter into a very different regime in which the material is undergoing large deformation which is enabled by the movement of dislocations through it. When this starts to happen, then all those little microstructural details in the metal such as grain boundaries, pre-existing dislocations, and fine precipitates become very important because they all act to block the smooth flow of dislocations through the metal. As a result, more stress has to be applied to the metal in order to overcome the dislocation barriers and make the metal plastically flow. That's why work-hardening and precipitation hardening (i.e., "age hardening") are so effective at increasing the yield strength, which is a measure of the stress required in order to make the metal plastically deform.
Elastic Properties (e.g., Young's modulus, Bulk modulus, Poisson's ratio) depend on the elemental composition of a metal but are insensitive to the microstructural details of a metal.
Plastic Properties (e.g., Yield strength, tensile strength, elongation at maximum yield) are sensitive to the microstructural details of a metal (e.g., grain boundaries, pre-existing dislocation bundles, fine precipitates) because these microstructural features can block the movement of dislocations through a metal which enable the metal to plastically deform.