I know a battery maintains a potential difference between it's terminals , let's say that the ground is at 0 potential for my question , then with respect to the ground both the terminals are at non-zero potentials.
This statement depends on what you mean by "ground". In electrical power systems "ground" refers to the earth, or "earth ground" because most systems around the world have a conductive connection to earth ground. Unless one of the battery terminals, or a wire connected to one of its terminals, is physically connected to the earth, one would normally consider the potential between either battery terminal and "ground" to be zero. However, capacitance always exists between any conductors, and that would include the battery terminals and ground or any other conductor. The amount of capacitance would, however, be undefined without knowledge of the position of the terminals with respect to other nearby charges, which can vary.
Now if I were to take a wire connect it's ends to ground and to either one of the battery terminals , shouldn't there be a current , at least momentarily?
Yes. Whenever you connect one of the terminals to something conductive (be it earth ground or simply a conductor), there will be electrostatic repulsion or attraction of the charges on the conductive body the battery is connected to, causing a momentary current (movement of charge) to or away from the surface of the conductive body.
The argument I have read everywhere is that the circuit is not complete so current does not flow but again we have a momentary current across a circuit having a battery and capacitor.What exactly is the explanation , for/against the current flowing in that wire.
You are correct. You don't need a complete, as in conductively complete, circuit in order for there to be movement of charge. A capacitor is a perfect example. No current flows through the dielectric between the plates of a capacitor. Yet if it is connected to a battery, the battery delivers charge to one plate, and receives charge from another. It will eventually cease when the voltage across the capacitor equals the battery voltage.
Bottom line, the explanation is whenever there is movement of charge, even if temporarily, we have by definition current.
Hope this helps.