There's a nice paper by Drs. Klaus-Peter Schroder and Robert Smith on the distant future of the Sun and Earth; it's available at the arXiv:
Table 1 in that preprint summarizes a number of parameters, but in simplified form the radii (in terms of the current value) at different times (given in billions of years) are:
Age Radius
ZAMS 0.00 0.89
present 4.58 1.00
MS:final 10.00 1.37
RGB:tip 12.17 256.
ZA-He 12.17 11.2
AGB:tip 12.30 149.
(hopefully that will render correctly.) For comparison, the current orbit of the Earth is 215 times the current solar radius. ZAMS is the zero-age Main Sequence, present is today, MS:final is the end of the Main Sequence, RGB:tip is the maximum size during the Red Giant branch, ZA-He is the start of core Helium burning and AGB:tip is maximum size during the asymptotic giant branch phase. After that the Sun will fade away as a white dwarf.
While there is 2.17 billion years between the end of the Main Sequence and the start of core Helium burning (which also marks the end of the Red Giant phase), for more than two billion years the Sun is less than ten times its current radius - it's only during the last 200 million years when the expansion towards the Earth's current orbit happens. This is plotted in Figure 1 of the preprint, which the radius of the Sun during the final three hundred million years.
So in the context of the Sun's overall lifetime, the expansion in the giant phase is extremely rapid. Of course, on our timescales it's a very long time...!