How can a day be exactly 24 hours long? The longest solar day of year is approximately 24 hours 0 min 30 seconds (occurs at mid to late December) while the shortest solar day of year is approximately 23 hour 59min 38 seconds. If I average out both of these I come up with average solar day of 24 hour +4seconds. Why then it is said it is 24 hours 0min 0 seconds exactly??
Wouldn't using a 24 hour solar day as a definition of day cause the offset of 4 seconds each solar year
 A: You can't calculate the length of a mean solar day just by taking the mean of the shortest & longest apparent solar days. That would work if the apparent day lengths varied in a simple linear fashion, but that's not the case.
From Wikipedia's article on the Equation of Time,

The equation of time describes the discrepancy between two kinds
of solar time. [...] The two times that differ are the apparent solar
time, which directly tracks the diurnal motion of the Sun,
and mean solar time, which tracks a theoretical mean Sun with uniform
motion.

This graph shows the cumulative differences between mean & apparent solar time:


The equation of time — above the axis a sundial will
appear fast relative to a clock showing local mean time, and below the
axis a sundial will appear slow.

To correctly calculate the mean solar day length you need to integrate the apparent day lengths over the whole year. (And you need to decide exactly how to define the length of the year, which is a whole complicated story in its own right).
There are two primary causes of the Equation of Time.
1. The obliquity of the plane of Earth's orbit (the ecliptic plane), which is tilted approximately 23° relative to the equatorial plane. This tilt is also responsible for the seasons.
2. The eccentricity of Earth's orbit, which causes the Earth's orbital speed to vary over the year. The following graph shows how these two components combine to create the Equation of Time.


Equation of time (red solid line) and its two main components plotted
separately, the part due to the obliquity of the ecliptic (mauve
dashed line) and the part due to the Sun's varying apparent speed
along the ecliptic due to eccentricity of the Earth's orbit (dark blue
dash & dot line)

Please see the linked Wikipedia article for further details.
I have further info, with better and additional graphs, on our sister site.
https://astronomy.stackexchange.com/a/48253/16685
This answer has more details, graphs, and some Python code:
https://astronomy.stackexchange.com/a/49546/16685
A: The 24 hours (exactly) definition of the mean solar day only applies if you use the UT1 time scale. As others have mentioned, the mean solar day is not the average of the shortest and longest apparent solar day, and you have to consider the Equation of Time to calculate the mean solar day by averaging all apparent solar days in one year. 
If you use the definition of a second based on the metric system, which is used by the atomic time (TAI), UTC and the terrestrial time (TT) time scales, the length of the mean solar day is not exactly 24 hours. The definition of the SI second, apart from being an atomic scale, is based on the mean solar day in 1900 as determined by Newcomb (in reality it corresponds to the mean solar day in about mid 19th century), when the Earth's rotation was faster than today. Today the mean solar day is slightly longer than in the past (when measured by an atomic clock), and the length of the mean solar day in TAI seconds is greater than in UT1. To correct for the difference, leap seconds are introduced 0-2 times per year in the UTC time scale based on observations of Earth's rotation, to keep UTC in sync with UT1 to less than 0.9 s. These corrections are publised in advance by IERS in Bulletin C. There have been 27 leap seconds inserted in the last 46 years, which equals to about 0.6 s per year difference between the length of the mean solar day between TAI and UT1, or about 1.6 ms per day (see Figure 1 and 2).
The length of the SI second depends on the location, and TAI is based on observations done by atomic clocks in various laboratories around the world ("UTC(k)") and corrected for the geopotential on the sea level. TT is a theoretical length of the SI second on the geoid, and as such is never known perfectly. Approximation of past TT is revised annualy by BIPM. In contrast, TAI and UTC are determined and kept fixed after about 1 month past the fact (published regularly in Circular-T by BIPM). Laboratory-specific UTC(k) time scales and the GPS time (based on the US Naval Observatory Master Clock) are known in real time. Other time scales include the geocentric time, barycentric time and the ephemeris time. The length of 1 s is slightly different between all of them, some of the differece is due to time passing differently based on location in the gravitation potential. For example, time passes faster in the solar system than on the Earth's surface by about 0.5 second a year.
Historically, time measured by the rotation of the Earth was the most accurate, and the mean solar day was assumed to be constant equal to 86400 s. This changed with the introduction of the ephemeris time, later superseded by the quartz clock and the atomic clock, which lead to re-definition of the second. They will deviate further as the rotation of the Earth slows down.
References:


*

*Urban and Seidelmann (2012): Explanatory Supplement to the Astronomical Almanac

*McCarthy and Seidelmann (2018): Time: From Earth Rotation to Atomic Physics

*BIPM time publications: https://www.bipm.org/en/bipm/tai/publications.html

*IERS Bulletins: https://www.iers.org/IERS/EN/Publications/Bulletins/bulletins.html

*USNO - The Equation of Time: https://aa.usno.navy.mil/faq/docs/eqtime.php

Figure 1. Excess to 86400s of the duration of the days, combined GPS solution, 1995-1997. From https://www.iers.org/IERS/EN/Science/EarthRotation/LODgps.html

Figure 2. TAI-UT1 and TAI-UTC. From McCarthy and Seidelmann (2018).
A: The average value of a distribution is not the average of its minimum and maximum. For example, the average value of (0,0,0,4) is 1, not 2. Earth's orbit eccentricity is not 0, nor the Moon's one, so your distribution of day duration is probably slightly asymetric, hence the 4 seconds discrepancy. Sum all day duration of a year, divide by the number of days, you will get a better value.
A: The original definition of an hour was 1/24th of a day, no matter how long the day was at the time. The durations you're quoting are only possible with the help of extremely modern definitions made possible by redefining the second (keeping the minute and hour fixed as 60 seconds, and 3600 seconds, respectively).
