Did the big bang start at ONE infinitely curved point (or a 4D curved ball shrunk to a point) with an infinite density of energy? I'm confused. In this question it is said:

The simple answer is that no, the Big Bang did not happen at a point. Instead, it happened everywhere in the universe at the same time.

How could it have happened everywhere in the Universe if the Universe hadn't come into existing yet? Or do I have a wrong image of the Universe at the moment of the big bang? Or did inflation began when the Universe already had a size (after $10^{-34}$ seconds, give or take)? Which begs the question, how did the Universe evolve from an infinitely dense and curved point until inflation took of? 
 A: It is called effective quantization of gravity:

In general the way singularities are treated in present day physics modelling is by using quantum mechanics, which always introduces an indeterminacy due to the the probabilistic nature of quantum mechanics. The coulomb  1/r potential is only used within the quantum mechanical equations but no test charge can  approach r=0 as  the solutions are quantized and a ground state exists. Thus one has stable atoms.
Quantisation of gravity is still a research topic for physics, as it is only effective models that use the concept, but the expectation is that gravity will join the other three forces and be quantized.  In the present day Big Bang  model there is no singularity but a fuzzy quantum mechanical controlled region of space time from which the universe of the present day evolved.
To answer your question it is a universe described in three dimensions of space and one of time, in the form of four vectors all the way, also into the fuzzy region where the singularity used to be in primitive Big Bang models.
