In my opinion, and this is very much non-standard, and there should and I am sure will be, other answers giving a more conventional view but you have hit the whole nail on the head in respect of there being some degree of absurdity to the notion of the big bang and the beginning of time.
What many fail to realise is that both time and size are themselves relative and as such, the notion of the Universe being the size of a pea is absurd at a time when neither a pea nor anything remotely resembling it exists, and if it did, wouldn't be meaningfully "pea-sized" anyway.
Where care is rarely taken then, is to make precise statements about that it means for the universe to be tiny.
What I interpret these statements to actually mean, is that the distribution of spaces within the universe was different insofar as the space within particles was much greater, in comparison with the space between particles, as a proportion of the total space, than it is now.
But the notion that metre back then had much to do with a metre now, is absurd.
What we can do, is to recover some meaningful notion of what we mean by expansion using a statement of the following form: "The distribution of space among matter shifts such that the proportion of space contained within particles decreases over time relative to the proportion of total space that is between particles."
Moreover, one should also take care with the other dimension of separation - time, which is more potent at reducing the notion of the big bang to absurdity.
According to certain of our basic models (if I later find the reference which was from John Baez, I will add it - for now I can find this discussion of a book by Roger Penrose, which speculates about time before the big bang), if we look at motion as we approach the big bang, going backwards, all motion gets faster and faster, and becomes a singularity at the beginning of time. Baez talked about this being a singularity, and if I remember rightly, possibly a shortcoming of our models. But what he failed to pick up upon in the blog he wrote (and I have seen no prominent physicist pick up on either) is that if true, this means the big bang is not necessarily the beginning of the universe.
If motion approaches infinite speed then things can happen infinitely fast - infinitely many things can happen in no time at all. With this knowledge, the theory of the big bang seen truly, only says that our concept of time breaks down around 13 billion years ago, but you can take any moment of your choice, arbitrarily close to the big bang and still always find more stuff taking place in what we call the "moments" before, than in the entire 13 billion years since.
It would seem therefore that rather than consider time to be linear, we should think of it as circular. Locally, change fairly unambiguously correlates with distance or separation, but but the further back we see, the more we approach an "event horizon" analogous with the horizon of a black hole, where it becomes difficult for us to separate events from each other in a linear fashion they resembles time as we experience it locally.
If I remember rightly there are tentative, but reputable theories from prominent physicists that compare the big bang with our emergence from a black hole and my personal opinion would be that this is likely a good analogy, if not description. My personal opinion is that the big bang is the moment we (the entire universe as we know it) as a collection of light, were emitted, and that we and everything around us recede at the speed of light relative to the big bang. Then the CMB is the weakest distant remnant of that event, that we can see.