When a potential difference is applied across a conductor, and if an electron moves from the negative terminal of the battery and reaches the positive terminal, then I want to know if the electron will remain at the positive terminal or will it again move toward the negative terminal through the battery?
Electrons that reach the positive terminal indeed remain there. The potential difference between the two terminals pushes electrons from the negative anode toward the positive cathode. When an electron reaches the cathode, it stays there to equalize the original charge imbalance between the two nodes. When electrochemical redox reaction sustaining the electron movement equilibrates, the motion will stop and the battery will "die."
As the diagram shows, the two terminals are connected by a "salt bridge." But the salt bridge is specifically designed to prevent electrons from flowing directly from the anode to the cathode. So the electrons can only flow through the circuit.
Battery is an electrolyte and positive ions can move in it rather than electrons. Similarly to the current of holes in semiconductor, directed flow of such ions complements the flow of electrons in the wire.
Here, electron-cation pairs are created at one electronde of the battary and recombine at the other (you may see that one electrode is destroyed while some material is deposed at the other by simply passing a current through the salty water).
Opposite flow of positive charges in battery/semiconductor is exactly equal to the direct current of the electrons in the conductor. You can think of positive charge moving in one direction as the current of negative charges moving in opposite direction.
Other answers just tell you that (a tiny pulse of) ion current is made first in the battery. It creates the voltage difference between anode and cathode. This causes electron current in the wire. The electrons stop at the positive electronde, thus, reducing the voltage. The battery restores the voltage by pushing more positive ions to that electrode. The flow of these ions is the current that you miss.
In an ideal electrochemical cell case, it should remain at the positive terminal. The salt bridge should only allow for the ions to flow.
However, practical batteries have other physical phenomenon that restrict the flow of current, and hence introduce this 'internal resistance'. There could be a wide variety of reasons depending on the type of cell used. I believe one of the most common reason is polarisation. But that too, shouldn't cause the electrons to flow through the cell.
To quote a good explanation from University of Illinois Physics Dept Q&A page:
EDIT: As to why there is current flow inside the battery: Electrons are not necessary for current to flow. The flow of ions does happen inside the battery.