If the universe is expanding, then at some time in the past, it must have started from a single point but why this point is not the center of the Universe. Just like the singularity of black holes is its center?
I'll answer your question with an analogy. Imagine a really small balloon, so small that it occupies a point. Now, imagine that the balloon is expanding uniformly outward from that point. Note that that central point is not part of the balloon. It's the same idea as to what happened with the BB. In this analogy, the universe is the surface of the balloon. The central point is where the universe started, where the BB happened, but that point is actually not part of the universe (it's not on the surface of the balloon). Incidentally, if you look anywhere on the surface of the balloon (anywhere in the universe), you'll notice that the surrounding area is expanding around the point you're looking at so it looks as though any point is the centre of expansion. That is also true of the universe. If you focus your attention on any point in the universe, it looks as though everything is expanding outward from that point so that it looks as though that point is the centre of expansion.
it must have started from a single point
This is a common misconception popularized buy the media. Imagine this grid:
Imagine each square getting larger. If you think about it, you will see that each point on the grid is expanding. The grid is the universe. Each point on the universe is its own "singularity".
The balloon analogy imagines the universe as a 2D surface expanding around a central point as it moves through a 3rd dimension of time. This may be the origin of confusion as in reality there is no 2D surface of expansion, like a wave front, but rather an expansion of 3D spacetime, wherein every point in space quite literally is its own central point from which the rest of the universe expanded. Hence the red shift of all distant galaxies, as would be the case regardless of your location. This is not as straightforward to illustrate, nor as easy to comprehend, as the balloon analogy, but we don't live on a balloon, we live within an expanding 3D space!