The deeper we look into space, the more primitive the structures are. Since we can see really primitive galaxies like Quasars, it's reasonable to think that there are even more primitive structures beyond Quasars, and eventually if you look deep enough, there would be nothing to see, because no structures, or even atoms, have formed so far in the past. The question of whether the universe is finite or infinite seems to be answerable this way, because clearly the universe must be finite if there is a radius beyond which nothing exists. So why is this question still not settled?
The reason why the question is still unresolved is partly given by what you said. The further we look, the further back in time we go, until we get to a point where we see the beginning of the universe (strictly speaking we can only see as far back as the cosmic background radiation allows us to). This limits the distance we can see. But this does not mean the universe only extends to this distance. Imagine for a moment that light propagated at infinite speed. Then we would have been able to see to the end of the universe. So the limit of what we can see is set by the finite speed of light and not by actual size of the universe.
This discussion will remain up in the air forever because the answer is abstract and uncertain as the question is about being infinite. Infinity is something beyond the grasp of our mind.
However, as a tool for practical problems we can assume the universe to be infinite. For example, we discuss infinite rods and sheets in electrostatics which aren't actually infinite but are infinite in the way we define the situations. For an infinite rod, for instance, if we are very close to the rod as compared to the rod itself then we can use the approximation that the rod is infinite (as the upper and lower end of rod are not in our field of view from that close distance). Thus we can see that infinity is something to be defined relative to some other thing.
As we have no known way of getting to the boundary of universe which is probably beyond the arena in which microwave background allows us to see, we can say for all practical purposes that the universe is infinite.
Physicists are of the opinion that universe is infinite because they have figured out that the galaxies and all parts of the universe are moving far apart from each other and this movement is accelerating as well. This also leads scientists to think of a Big-Bang like beginning for the universe.