How can a Population III star have a mass of several hundred solar masses? Normally the limit is about 100 solar masses.
I think there are really three questions that need to be answered for this to make sense:
- is there a "normal" limit to how large a star can be?
- how can population III stars form with such large masses?
- how can population III stars retain their large masses?
An answer to the first question is tricky. We expect large stars to be rare, and the largest stars to be the rarest. On top of this, they'll lead the shortest lives. Getting observational constraints has thus been tricky. There might be a limit to the amount of mass that is available to turn into stars when they form. As for the "normal" limits on the masses of stars, most (as far as I know) involve around pulsational instability. But the recent discovery of massive stars in and near the cluster R136a suggests that stars with masses over 150 solar can form even in material that has a non-negligible metal content. So whether there is a "normal" limit is open question.
The second question is much better understood, thanks to a lot of numerical work. Tom Abel recently wrote an article for Physics Today that summarizes current understanding of pop III star formation. Basically, the smallest amount of gas unstable to collapse under its own gravity, the Jeans Mass, increases with temperature (like T3/2). So the cooler the gas can become, the smaller the fragments we expect to see. What determines how cool the gas can become? The atoms and molecules that radiate within it, and whether this radiation can escape. In metal-polluted gas, various molecular and atomic lines allow the gas to cool to tens of K. In metal-free material, the most effective coolant (in terms of the low temperatures it can achieve) is molecular hydrogen, which will only cool to around 200 K. This is a higher temperature, so we expect more massive fragments. This is a gross simplification! The situation really involves complex dynamics, shock formation, and all sorts of other stuff. Even the question of whether or not molecular hydrogen can form is contested.
Finally, if a massive pop III star formed, would it keep its mass? We know that the some massive stars in the local universe, like Eta Carinae, are violent beasts. This kind of episodic, pulsational mass loss could be present in Pop III stars, but since such mass loss is so poorly understood, this is often ignored. More generally, we expect that the metals in the atmospheres of massive stars absorb enough of the radiation created inside the star to be driven away in a wind. Again, there aren't any metals in metal-free gas, so we expect this effect to be much smaller in Pop III stars.
So, we expect Pop III stars to be larger because there is more gas available, because the gas fragments less owing to its higher temperature, and because we don't think the stars lose as much mass as modern stars do. And, we aren't even sure that there's a limit on how massive stars can be in the first place!
The key as I understand it is metallicity. The Big Bang produced virtually nil above helium, so Pop III stars and their ancestral HI clouds had almost no metals. The forest of emission lines produced by even a tiny fraction of metal atoms acts to increase the cooling efficiency of the cloud enormously. At the extreme rate of cooling of modern clouds, protostars have very little time to accrete mass, so the clouds break up into lots of smaller stars. Primordial clouds, however, would have cooled so much more slowly that they could maintain structural integrity long enough to form fewer, much more massive stars.