Electromotive force EMF is defined as:
                   "The potential difference across the terminals of a battery or dry cell when it is giving no current to the circuit."
So, if current starts to flow from the battery, will the potential difference across the ends of the battery not called EMF....?
 A: Though the terminology of electromotive force (not a force) is actually the external work done per unit charge, there's another definition (which you've already mentioned) in current electricity. So, It will not be called EMF if the circuit is closed. It's then called the voltage. Both are similar where EMF is the cause and the latter is the effect.
The problem with EMF that it's measured in an open circuit is because there's always an internal resistance associated with a cell which depends on the electrolyte inside. Though the current flows from positive terminal to negative terminal in external circuit, there's a necessity for it to flow from negative terminal to positive terminal inside the cell. The internal resistance $r$ increases with the age of cell.
If $E$ is the EMF of the cell, $I$ is the current, $V$ is the voltage drop across the circuit and $R$ is the effective resistance associated with the external circuit, then the internal resistance could be measured by including a resistance in the circuit and comparing the voltage drop across them. The voltage drop across the resistor would be $V=E-Ir$ and the internal resistance could be calculated as $$r=\frac{(E-V)}{V}R$$
