You could use a setup similar to a Galileo thermometer, i.e., have a body with an (average) density which is below the density of water at $4^\circ\mathrm{C}$ but above that at e.g. $0^\circ\mathrm{C}$ and $8^\circ\mathrm{C}$: It would swim at $4^\circ\mathrm{C}$ and sink at both higher and lower temperatures. This actually works, as witnessed by the Galileo thermometer; if you use a design for your floating body similar to the Galilean thermometer, it should be possible to tune the average density quite precisely.
It seems that the density difference between $0^\circ\mathrm{C}$ and $4^\circ\mathrm{C}$ is about $1/3$ of that between $20^\circ\mathrm{C}$ and $21^\circ\mathrm{C}$, so it should be possible to resolve this difference. (Source.)
Of course, the same caveat as in my comment above holds: You need to assume that the thermal expansion coefficient of whatever body use use is negligible as compared to that of water.