This effect is called the elastic limit: https://en.wikipedia.org/wiki/Yield_(engineering)#Definition
Elastic limit (yield strength)
Beyond the elastic limit, permanent deformation will occur. The elastic limit is therefore the lowest stress at which permanent deformation can be measured. This requires a manual load-unload procedure, and the accuracy is critically dependent on the equipment used and operator skill. For elastomers, such as rubber, the elastic limit is much larger than the proportionality limit. Also, precise strain measurements have shown that plastic strain begins at low stresses.
As stated in the Wikipedia article, as the rubber band comes under higher degrees of stress, it will experience a stretching (strain) which it will recover from if the strain does not surpass the elastic limit.
Also remember that the strain experienced by the material is time-dependent, so if your rubber band is under constant stress (stretched around something for long time) it will be more deformed than if stressed the same amount for only a moment.
For more information on strain accumulating over time: https://en.wikipedia.org/wiki/Creep_(deformation)
Creep (deformation)
The rate of deformation is a function of the material properties, exposure time, exposure temperature and the applied structural load. Depending on the magnitude of the applied stress and its duration, the deformation may become so large that a component can no longer perform its function.
All materials will experience this creep. If a material can be strained by stress (can be stretched), it can be deformed by experiencing a large enough cumulative strain.