Potential energy is the work done by the particle against a conservative force. Isn't it? when a charge is fixed on a free space and then when we bring another charge from infinity to a distance r from the first charge a work was done against electric field force so it gains some potential energy. But as of now the first particle also has a potential energy. Since no work was done on the first charge against the electric field force which was given by the 2nd charge as it didn't move how can we tell that the 1st charge also got a potential energy?
In your first case you said you were bringing a charge B to the first charge lets say A. Here your reference is A thats why u say you are bringing it to A. Think of it the other way, put you reference on B then your perception would be you are bringing A to B and now A will have a Potential energy.
Thats how it works both ways.
And Finally i would like to correct you, all this while you were talking about Potential Difference and not potential energy check out the difference for yourself.
The potential energy is related to a system not to an individual charge. So when you bring one charge towards another charge the system of two charges gains potential energy.
What this means is that any work which is done by either or both of the charges will result in the loss of potential energy of the system (both charges).