From the books that I read, the discussion and the formulas related to the Fermi Level are always relative to the energy level of Conduction/Valence Band, or Fermi Level in intrinsic semiconductor.
Let's assume that the measurement is done at exact temperature.
Fermi level characterizes the filling of the energy levels, e.g., the concentration of electrons is given by $$n = \int dE D(E)\frac{1}{1 + e^{-\beta(E-\mu)}},$$ where $D(E)$ is the density-of-states and $\mu$ is the Fermi level. As you see from this equation, shifting the Fermi level will make the occupations of all the energy states change and this will change the electron concentration and other parameters. The only way to have everything remaining consistent is to treat the Fermi level as any other energy, i.e. it is measured in respect to the same origin as all other energies.
What might be a possible source of confusion here is that Fermi level is not the same thing as Fermi energy, which is the energy distance between the bottom of the conduction band and the last occupied state, as I discussed here.
Experimentally, the most straightforward way of measuring this is photoemission spectroscopy. But it is often difficult to measure the position of the Fermi level (chemical potential of the electrons) in an intrinsic semiconductor. The charge carrier distribution is small, and the Fermi level is often pinned by surface states. Also, low conductivity may cause charging.
So most of the comparisons of Fermi levels in doped and intrinsic semiconductors are theoretical.
