The answer is that this is an extremely complicated situation and a lot of effort has been put into designing speaker systems that will do this. It's not helped at all because the whole thing is surrounded by golden-eared hi-fi nonsense, and you need to distinguish between this and the actual truth.
I apologise in advance for the hand-waving naure of this answer: perhaps a proper audio engineer could point out the many oversimplifications and untruths in it! It's also much too long and has too little physics: sorry!
John Rennie's answer is correct but not actually very useful. For instance if you took a speaker with a cone 1cm across then, although you can drive it at 80Hz you're going to have a hard time hearing the sound it makes (you will probably hear all sorts of harmonics, but once you filter those out there's not going to be very much fundamental there).
But you would have no trouble hearing that same tiny speaker driven at 1kHz (even factoring in the different sensitivity of your ears at these two frequencies).
So cone size does matter. But in fact other things matter as well: in particular the enclosure the speaker is in matters a lot: there's a reason that loudspeaker enclosures which have good low-frequency response are large.
A rather unprecise but (I think) useful way to think about the situation is to think of a speaker as a flat bit of something rigid which you are moving perpendicular to its plane, pushing it by a rod attached to the centre of the board, in a sine wave (for simplicity) at various frequencies.
Let's take three cases:
Let it move at an extremely low frequency (how low depends on how big it is). Well, clearly, it's going to push on the air in front of it (and behind it, as it moves back). Equally clearly the air will just leak around the edge of the thing, and almost nothing will get coupled into the air far from the board.
Crank up the frequency. As you do so, the leakage of the air around the edge of the thing becomes less, because it takes some time for the air to leak, and it doesn't have time any more: as soon as its started leaking around to the back, the wretched board is moving the other way and it has to start leaking back. So more energy from the board gets coupled into sound in the air.
Crank up the frequency much further. Now there is another problem: although the board is rigid, it's not that rigid. So now, as the centre of the board moves forward, the edges haven't yet realised this, and are still moving backwards: there are, in fact, waves propagating through the structure of the board. The end result of this, far from the board, is that you can 'hear' both the forward and backward moving bits of the board, and you get much less sound (and complicated interference patterns as well, which don't help at all).
So the end result of all this is that two things matter:
- the size of the cone of the speaker, which limits HF response as the cone stops being rigid at very high frequencies;
- the size of the 'baffle' around the cone -- the enclosure it's in -- which limits the low frequency response of the system.
What I've failed to do here is to describe two things: why can't a small speaker in a big baffle produce a good low-frequency response, and why are baffles in practice fairly small.
Well, let's consider a small speaker in a big -- infinite in fact -- baffle. And consider driving it at some very low frequency. Now, even though air can't leak around the baffle, it can leak along it, and the distance it leaks along depends inversely on the frequency, so the amount of air leaking goes at least like the inverse square of the frequency. What's essentially happening is that the whole baffle is becoming a speaker, and in order to drive it, you need to pump a great deal of air in and out of the speaker. Well, you can do this, but in order to do it, you need the speaker to have a very large 'throw' -- it needs to move a long way forward and back. Such large-throw speakers are hard to design because their suspensions must both support these large displacements and be reasonably linear. Traditionally it has been much easier to make a much larger diameter cone with a much smaller throw, which moves the same amount of air.
(More recently, people do seem to use much smaller cones for low frequencies: they're much less domestically-hostile of course. I think that better materials and probably also servoing the thing have helped here.)
The baffle size thing is complicated. One trick is that if you completely seal the enclosure (so the speaker is in a closed box) then you have made what is effectively an infinite baffle -- and this is what such enclosures are called. Infinite baffles are hard to understand (at least for me), but I think their low frequency response is typically limited by the fact that they are sealed, so you end up having to compress the air inside them as you move the speaker cones. But I don't understand them.
A simpler kind of enclosure to understand is what's called a 'ported' enclosure. From above you can see that what limits the low frequency of a finite baffle is the linear size of it: how long it takes for the air to find its way around the edges. Well, you can make a baffle which is much bigger in linear size than it actually is, by connecting the front of it to the back by a long, folded pipe: it's the length of this pipe -- or port -- which defines what the effective linear size of the baffle is. I think most modern speaker designs are ported enclosures.
Another point is that, of course, the whole system is designed together: the speaker cones which sit in the baffle (enclosure), and the electronics driving them (traditionally the crossover, a passive device, but more recently the active system driving them) are all a whole system, and all the bits of it alter the response. And, of course, it is generally all designed not to be some beautiful linear thing (despite what people might have you believe) but to sound good, which means it needs to sound like people want it to sound. I have two pairs of ex recording-studio monitors, which generally are designed to be as neutral as possible because that's what you need in studio (no point in mixing something which only sounds good on the monitors you have) and which have enormous (15in) speakers in cabinets you can't lift, and people are often disapointed that these huge monster things have 'so little bass': well they actually have really ine bass, they just don't have the enormous bump below about 100Hz the way most speakers have to make them sound impressive.
Finally I'll say again that this is complicated: the frequency response of speaker systems is really not easy to understand, especially when they are dumped into some completely uncontrolled listening environment. This is a hard engineering problem. The only thing that it is completely safe to say is that (almost) all of the hi-fi nonsense is just that.