What does the concept of an "infinite universe" actually mean? When physicists talk about the universe being infinite, or wondering whether it is or not, what do these two options actually mean? I am not interested whether the universe is infinite or not, I am interested in what are the two options actually looking like.
Does an "infinite universe" mean infinite space, but when you go far enough there isn't any matter anymore, just more of infinite empty space in that direction?
Or does an "infinite universe" mean infinite matter, where it doesn't matter how far you go, you will find infinite stars all along the way?
In short, is the "infiniteness" meant to apply to space or to matter (which then would also include space of course)?
 A: Generally when physicists talk about the universe being finite, they are talking about the existence of an upper bound $R$ on the distance between any two points in space.  Such an upper bound could arise in several ways - perhaps the universe has an edge - a boundary which cannot be crossed - or perhaps the universe has the topology of a 3-sphere, and so if one travels sufficiently far in any direction they would eventually return to their starting point.  A spatially infinite universe is one which does not have this feature - given any real number $M$, there exist two points in the universe which are separated by a distance which is greater than $M$.
A finite universe of course can presumably only host a finite amount of matter.  An infinite universe could in principle host either a finite or infinite amount of matter.  Mainstream cosmology generally assumes the cosmological principle, which states that on sufficiently large scales the distribution of matter in the universe is homogeneous; in such cases, an infinite universe would have an infinite amount of matter in it, but of course this principle may not be accurate.
A: Maybe it is best to mention the three major possibilities of "universal shapes" that exist today.
Let me start by saying that cosmologists envision the universe as a four-dimensional spacetime continuum that can exist in a variety of shapes as long as the shape conforms to the field equations of Einstein's general theory of relativity.
Hornet-shaped spacetime.
And let me start with a very strangely shaped universe. The funnel formed spacetime. Look here or here. This is a hornet-shaped (hrnet=curved) universe. That is if we look at two space dimensions because we obviously cannot see a curved three-dimensional space. This universe allegedly explains observations made on the universe. This is a finite universe. Here's a picture (note that the points of space where the ship turns back inward are problematic):

A spherical spacetime.
The most simple and most common form. This is the "balloon universe", due to the fact that the two-dimensional space part can be viewed as a balloon shape, which can be viewed to represent a curved form of a two-dimensional space.
This universe can be finite or infinite in spatial extent. And these spacetimes can have zero, negative or positive curvature.
Zero curvature corresponds to a spacetime that will continue to expand forever and this expansion will stop "after" infinite time, while its size is infinite. The balloon will stop expanding at infinity and have an infinite size.
The positive curvature represents a universe that expands and after a finite time, this expansion will stop. This universe obviously has a finite size. The balloon grows in size and will contract after some time (spacetime generally cannot stay motionless).
The negative curvature represents a spacetime that will expand at an ever-increasing rate. The ballon starts expanding and will grow in size faster and faster. It looks as if our own universe is (in these days) growing with this last feature.
In our universe, inflation has been assigned to a newly formed balloon spacetime. Initially, the universe expanded at an incredible rate and basically, the whole universe was formed during inflation. Since the end of inflation, the universe has only grown "only" three times as large as in the beginning. So what we see today is just a very tiny fraction of the whole. Behind the horizon the is a vast space.
The three universes are also referred to as flat, closed, or open:

The torus-shaped universe.
In 1984 this model was proposed as the [three-torus model]. 5This universe has a two-dimensional space equivalent of a torus. It has positive as well as negative curvature and can be finite or infinite (the infinite size will be reached after infinite time).

One can invent more shapes, of course, but they have to pass the tests of observation. I'm not aware that another form as the three mentioned is present though.
It is generally accepted (i.e., by most cosmologists) that our universe has a balloon=like topology. The amount of matter in these universes can be finite or infinite. In closed universes, the amount of matter will be finite while this amount is infinite in (asymptotically) flat or open universes.
What about time? Is spacetime eternal? That depends of course if there was something before the big bang (when the spacetimes started evolving very small size). For example, according to Smolin, there was a contracting universe present before the big bang (likewise he proposes that upon entering a black hole there will follow an exit to another spacetime, where there is a white hole present which ejects all that has fallen in from our own spacetime). It is generally claimed though that spacetime had a beginning in time and dependent on how much matter is in it and on the expansion speed, the universe will be there forever or not. It is thought (based on observations) that our universe is nearly flat and will expand forever.
So, what does the concept of an infinite universe mean? In the context of spacetimes that might be clear now: a universe is infinite if the spacetime has an infinite extent. Infinite space goes hand in hand with infinite time. The amount of matter contained in these universes can be finite or infinite.
A: When cosmologists talk about an infinite Friedmann universe, they mean one with infinite spatial volume, finite and uniform matter density (averaged over cosmological scales), and thus infinite matter. There would be galaxies as far as you care to go, even if you could go far beyond what we call the “cosmological horizon”.
A: When you ask about an aspect of the universe as talked about by most cosmologists, you need to take into account the common general assumptions about the universe. One of the most common assumptions is that the universe is approximately the same everywhere taking an average of such things as average matter density or radiation density in a sufficiently large volume. The is called homogeneity. A second assumption is that the universe looks the same with respect to a sufficeient large scale in all directions. This is called isotropy.  With the assumption of homogeneity, an infinite universe is expected to have in a sufficiently large volume the same stuff and dynamics everywhere, just like a finite universe does. Different homogeneous universe models have different conditions at different times. It is a bit tricky do decide how to determine what it means for two different volumes in the universe to be at the same time. The simplest way to think about it is that if they have the same densities, then they are at the same time. Some of the other answers talk about rather strange shapes, like a hyper-torus. So far I have never been able to locate a description of how a torus can be homogeneous and isotropic.
A: It doesn't make any sense to talk about spatial scale/distance without referring to some objects within space. Or have you ever used a ruler to measure the distance of two space points in empty space? This pretty much answers your question.
