If there is acceleration of the test charge then the charge would have kinetic energy along with the potential energy.
Potential is defined as the work done by the external force per unit charge against the electric field of the reference charge (the source of electric field). So if the charge is accelerating then it means that the external force is not equal to the force due to the electric field. Thus, we won't get the correct potential this way.
While calculating the potential due a charge, we only consider the change in potential energy of the test charge when bringing the charge from infinity towards the reference charge.
We only want to consider the potential energy per unit charge to calculate the potential, so that if we wish the find the potential energy of an another charge, we just have to multiply the charge of that particle and the potential of the point where we want to find it's potential energy.