I'll try to answer this with limited physics and General Relativity vocabulary, hoping that my underlying understanding and interpretation of it is not severely distorted....
Time on earth would have elapsed much faster compared to the clock that the astronaut carried to his destination closer to a black hole. Meaning, the astronaut would have aged lesser when he returns. However, the space itself is consistently expanding. This would imply that both — the observer on earth and the astronaut — would see the galaxies receding away. The difference though would be in the "rate" at which they'd find them drifting away.
Both will have the same basis, before the astronaut's journey starts, to believe that the universe is 13.8 billion years old. Now, from the point that the astronaut's journey begins, they'd start measuring everything related to time, length, distance and speed differently — including their own and universe's age — in their own reference frames. Assuming that he returns back to the earth's reference frame after a few billion earth years (and assuming that the earth is still habitable), his calculations would show that the universe hasn't aged as much as it has for the then residents of earth. Even though both would agree that the universe has since expanded, they would disagree on the rate at which they "saw" it expanding.
There is one very serious catch though... The distant galaxies are moving away faster than the speed of light and at some point in the future there will be nothing else to see in our "increasingly expanding" sphere of the observable universe, aside from our own galaxy. So, if the astronaut, in his venture, goes far beyond into the space — into some other galaxy — and stays there in the vicinity of a massive stellar body for a "while", the expanding spacetime could even ensure that his observable universe would nowhere have Milky Way / Solar system / earth in his sphere of influence and sight, putting us and him out of each other's reach, forever; even if he had the means to travel with near-light-speed necessary to commence the journey in the first place.
Hopefully this describes the situation somewhat realistically. I also hope that if I've been incorrect, then someone would point out, comment on and refine the misjudgements, as appropriate.