The main reason is due to the fact that the prism refracts light in such a way that the "blue" part is more spread than "red" part. So that overall the energy hitting the thermometer is greater in the infrared and red part than on the blue part of the spectrum.
Edit: I have just seen your edit. You're right. There you can see the details.
Here's a quote from the website:
The answer turns out to be the experimental design, and a failure to correct for refraction. In Herschel's setup, sunlight is refracted by a prism. The index of refraction of course must vary with wavelength so that the sunlight would be dispersed into its various colors. If the index of refraction varied linearly with wavelength, Herschel would not have needed to correct for that variation, since the wavelengths would be uniformly spaced along his table.
However, since the index of refraction varies non-linearly with wavelength, the wavelengths will not be uniformly spaced along Herschel's measuring table. The actual spacing of the wavelengths versus distance along his table for an incidence angle of 45° from air into glass shows that the infrared region is much more highly concentrated than optical wavelengths. (The plot shows the spacing along the spectrum divided by the distance from the prism. Hence to get the actual spacing in cm or inches, multiply by the distance from the prism in cm or inches.) The relative concentration factor is shown normalized to 0.60 micron.
The net result is that Herschel's observed "temperature" should then peak in the infrared.
The website also provides some graphs and in one you can see that the red part (~650 nm) of the spectrum is about 3 times more concentrated than the blue one (~450 nm), while the infrared part is even more concentrated.