This has actually been a very important problem in the history of measurement and physics more broadly. The solution was an iterative process of increasing internal consistency and precision of measurements. The story has been extensively covered in Hasok Chang's fascinating Inventing temperature, which is my source for the text below.
A Twitter thread with some highlights from the book: https://twitter.com/zerdeve/status/1099815636652371968
1. Perception
Very roughly, early thermometers (1600s) were calibrated to our perception of heat - anything that measures temperature, should give higher value on a sunny summer day than on a winter night, its value should increase when placed closer to a flame and decrease when moved away etc. Any device that passes a lot of such tests is likely to be monotonically related to temperature. Note that already we can move beyond just sensation - if we know a device was well calibrated to perception on many ocasions and then we find our perception contradicted by the device, we may come to believe that this time it is our perception that is at fault.
2. Fixed points
Then people tried to establish "fixed points", i.e. events that reliably happened at the same temperature. The idea is that you take a device that seems to agree well with subjective perception of heat and measure the event multiple times. If it is truly fixed, you should measure the same response everytime. You can then try with a proto-thermometer based on a different principle. Many fixed points were proposed, but same later failed that criteria (e.g. bodily heat was tried as a fixed point, but found varying when measured with precise - but uncalibrated - thermometer). The freezing and boiling point of water was first used as fixed points around 1690.
Here the iterative nature of the process reveals itself again. Once we establish a fixed point with decent confidence, we can use that fixed point to validate thermometers and rule out devices that don't yield sufficiently fixed values at fixed points. We can then use those better devices to better validate fixed points, ... If we do this right, we will find better and better internal consistency in our methods.
As an example of the iterative nature, boiling point of water was a pretty strong candidate for a good fixed point initially (after it was realized you need to control the pressure). But it turned out that it is pretty fixed only if you are not too careful. If you take care to avoid all impurities etc., you can easily superheat the water beyond 100°C without boiling. So thermometers calibrated to boiling point were then used to disprove the boiling point as a fixed point. The steam point (temperature of saturated vapour) at atmospheric pressure is in fact a bit more consistent and was preferred since ~1780s.
3. Linearity
The next challenge was then find a linear scale of temperature. Below we see see measurements made with three different thermometers based on expanding liquid in a tube, calibrated to freezing and steam point of water as 0/100°C. The distance between 0 and 100 was then evenly split into 100 segments:
Obviously, there is a problem. One answer was mixing: if we mix equal amounts of say 0 degree water and 50 degree water, we should get 25 degree water. In this experiment, mercury clearly gives superior results to alcohol, water or other liquids.
However, mixing implicitly relies on the fact that the specifc heat (heat required to increase the temperature of 1 unit of volume by 1 degree) is the same for all baseline temperatures. This assumption was considered suspect (e.g. because the volume of liquid water changes slightly with temperature, so why wouldn't the specific heat change as well), and thus no full agreement on how to divide the scale between the fixed points was possible.
Enter gases. In early 1800s people have established that gases expand with temperature and this expansion is extremely regular for many different gases. This led to interest in air-based thermometers, but theoretically, there was still no reason to believe that the measurements of air thermometers are linear. Different theories of heat led to different conclusions.
The impasse was resolved in ~1850 by Henri Victor Regnault with a simple idea, high experimental skill and big budget: measuring the same substance (e.g. the same bath of oil) simultaneously by multiple thermometers should give the same values across the whole scale (e.g. as you slowly heat the oil up). So any thermometer technology where multiple devices do not agree across their whole scale is suspect. When such experiments were performed with high precision, it was determined that even relatively minor variations in the construction of a mercury thermometer (e.g. different glass used for the tube) meant that two thermometers that agreed on 0 and 100 did not agree at other points of the scale. However, even quite different models of constant-volume air-based thermometers (different volumes of gas, different gases) did in fact agree quite closely at all points of the scale. Air was thus clearly superior and remained the candidate for a good linear scale.
Finally, as we got to understand thermodynamics and got a good grasp on what heat actually is, we could justify thermometer measurements and the linearity of temperature by that (very succesful) theory. But note that this happened after the events described above and would likely be much harder to achieve without precise thermometers already available.