Measuring the Voltage of the 2 sides of battery with reference to the ground I know potential difference or voltage is meaningful only if we have reference. So I took my reference as the ground and tried to measure the voltage of the 1.5 V battery.
First, I measured $$V(\text{minus side of the battery}) - V(\text{ground})$$ I measured 0 V, or near that.
Then, I measured $$V(\text{plus side of the battery}) - V(\text{ground})$$ I measured the same as the first.
Finally, I derive 0 V as the voltage of the battery.
What is the problem here? Shouldn't I get 1.5 V as the result?
edit: my reference is dirt, and yes I actually stuck  the reference probe (Black) into the dirt
 A: The issue is that your voltmeter is not perfect. The way a voltmeter normally works is that it consists of a very high resistance (megohm range) between the leads. Then, instead of measuring the voltage difference directly, it measures the current through that path and and solves for the voltage by Ohm's law $V=IR.$
But it only takes a very small amount of charge to change the voltage of an isolated object. A battery maintains the voltage difference between its terminals, but the "absolute" voltage of the terminals (relative to some isolated point) can be changed by adding/removing some net charge to/from the entire battery. (Note that a battery in normal usage in a circuit supports a current, but the net charge of the battery doesn't change with time.)
So what happens is that as soon as you connect two isolated conductors (here the battery and Earth) with your voltmeter, the voltmeter conducts a small current between them as part of its normal operation, but because the conductors are isolated and one of them is small, the tiny charge that flows is enough to quickly change the voltage being measured. In particular, the voltage quickly goes to zero.
Measuring voltages between isolated conductors is basically the realm of static electricity. There exist electrostatic voltmeters that can measure voltages without requiring currents that will alter the measurements. However, static electricity usually involves hundreds/thousands of volts, so such a voltmeter may not register the puny voltage of your battery. I suspect finding components good enough to perform your experiment as stated will be hard.
(Addendum based on Dale's answer: if the connection between your voltmeter and ground through earth is also bad, then effectively your "ground" is another "small conductor" consisting of the bit of soil just around your lead. It now takes even less charge to destroy your measurement!)
A: 
I know Potential difference or voltage is meaningful only if we have
reference.

It is potential that is meaningful only if we have a reference. Potential difference does not require a reference. The battery voltage of 1.5 v is the potential difference between the battery terminals.
So if by "ground" you mean that you are physically connecting one of the leads of your voltmeter to the earth or some metal part that is grounded, and the other one of the battery terminals, you will not measure 1.5 volts because the other terminal is open (isolated). To measure the battery voltage, you need to connect your voltmeter leads to the terminals of the battery.
Hope this helps.
A: There are a couple of issues with your measurement.
First, simply sticking a little voltmeter probe into dirt is insufficient to “earth” or “ground” your circuit. Typically a grounding rod is around 2 m long and needs to be driven into the ground securely, and if the soil is dry or rocky then you may need to use multiple rods. It is rather exhausting, but you need a fairly large contact area to get a good electrical connection with the earth.
Second, you need to measure the voltages in the same circuit. You measured 0 V in one circuit and then 0 V in a different circuit. The difference is not meaningful. The proper way to do this measurement is to use two voltmeters and measure both simultaneously.
Third, recognize that the voltmeter has a finite impedance. Typically it is very large in comparison to standard circuits, so connecting it only changes the circuit a little. But when the rest of your circuit is an open circuit then the voltmeter itself becomes the low-impedance path. Then the voltmeter substantially alters the circuit.
A: You start off with the battery and the ground.
The potential difference across the battery terminals is fixed at $1.5\,\rm V$ across its terminals.
Without taking a reading ask yourself, what is the potential difference between each of the terminals and the ground?
You should soon come to the conclusion that with the information given it is indeterminate and for example, the negative terminal could be at a potential of $-1\,\rm V$ relative to the ground and the positive terminal could be at a potential of $+0.5\,\rm V$ relative to the ground or $\pm 0.75 \,\rm V$ or any other combination of potentials as long as the potential difference between the two terminals id $1.5\,\rm V$.
All of this assumes that the ground is an equipotential surface.
So what happens when you connect your voltmeter from the negative terminal to the ground?
If there is a potential difference between the negative terminal and the ground and given that the voltmeter is a conductor, charges will flow until the potential difference between the negative terminal and the ground is zero and then no more current flows and the voltmeter reading becomes zero.
You can then surmise that the potential of the positive terminal relative to the the ground is $+1.5\,\rm V$.
Oh, that means that if I put another voltmeter between the positive and the ground I should get a reading of $+1.5\,\rm V$ on the second voltmeter?
No, that is not the case because you now have a series circuit consisting of two voltmeter, the battery and the ground and the voltmeters are measuring the voltage across their terminals.
If the ground, battery and connecting wires had no resistance and the voltmeters were of the same type then each voltmeter would show a reading of $0.75\,\rm V$.
