There are a few things to clarify:
the universe is not infinite in age, the big bang happened 13.8 billion years ago
the universe might be infinite in size, but what we mean by that is that the universe is practically endless
There are two ways to say if something (our universe for example) is infinite:
infinite spatial extension (or infinite 4D extension in our universe), meaning that it extends (for example as a sphere in 3D) infinitely, so its axes would be infinitely long, meaning it is unlimited in spatial extension
infinite meaning endless, that is, you can travel in space forever without hitting the edge (that possibly does not exist in our universe)
Now in our universe, the case is possibly 2., infinite, meaning endless, you can travel in space without hitting the edge forever.
Now the good example is the surface of Earth for an ant. The ant can travel on the surface of Earth forever without hitting an end, there is no end of road sign. Does this mean that the Earth has infinite spatial extension in space? No. The earth has a limited spatial extension in space (you could put/limit the Earth in a cube for example).
Yet, the Earth is infinite for the ant (who does not understand the bigger picture), who only lives in 2D on the surface.
Analogously, the universe is infinite for us, who live in 3D space, meaning, we can travel infinitely without hitting the edge.
Does this mean that the universe is infinite in age (in 4D in both directions, back in time and forward)? No.
Analogously, the universe is limited in 4D, so that the dimension of time is limited in one direction, back to the big bang, and it does not make sense to go back before the big bang, there is no such thing as before the big bang.
The very important thing is to understand that the infinity (in this case endlessness) of space does not require infinite time.
Based on the comments, the question is rather why we see the CMB if the universe is infinite in size and all the radiation has diluted.
Now some people might make you confused asking questions like "why is the nightsky not dark then?"
It all comes back to the time dimension. The reason we still see light coming from other galaxies, and we see the CMB is that the universe is not infinite aged. The universe according to our currently accepted theory, the SM, is 13.8 billion years old.
To have what you suggest, that the CMB radiation has all diluted (and analogously the night sky is dark), the universe would have to be infinite aged, meaning the universe would have had existed for an infinite time.
This is not the case, CMB, radiation, just like other photons (like visible) are coming from all direction in the universe, and the farthest we see could have been emitted only as much as 13.8 billion years ago (but again this is for the whole universe, not only for the observable).
If we would see no CMB radiation (and no visible photons either coming from galaxies), that could mean two things:
the universe has existed for infinite time
the expansion's rate has exceeded a certain limit, where every other galaxy is so far, that no photons (not visible, nor CMB) will ever reach us