For instance, if the work done by the electrostatic force due to a system of charges in bringing a charge of 10 C from some reference point B to a point A is 100 J, does this always imply that potential of point A is -10 V? I mean I could do the same amount of work (100 J) to bring the same charge from a point at 20 V to a point at 30 V, right? So, if the potential of a point is basically the change in P.E. (-WD by E.F.) per unit charge then the potential becomes independent of the reference point but that means bringing a charge from 20 V to 30 V, would make the potential of the final point as 10 V? Clearly, 30 V != 10 V. I am really getting confused here. Where am I wrong?
The best you can do is to find the potential difference between position $A$ and position $B$ ie the potential of position $A$ relative to the potential at position $B$.
The potential difference between position $A$ and position $B$ is independent of the position which you have chosen to have a potential equal to zero which you might call the reference position.
The potential at $A$ does depend on the position which you have chosen to have a potential equal to zero.