What you're "missing" is that you're basically right! The Gregorian calendar is slowly losing some accuracy. I've previously heard the error quoted as about 3300 yr, which lines up with your measurement because 1 day / 26.75s $\approx$ 3230.
From Wikipedia's Leap Year article:
The marginal difference of 0.000125 days between the Gregorian calendar average year and the actual year means that, in 8,000 years, the calendar will be about one day behind where it is now. But in 8,000 years, the length of the vernal equinox year will have changed by an amount that cannot be accurately predicted (see below[vague]). Therefore, the current Gregorian calendar suffices for practical purposes, and the correction suggested by John Herschel of making 4000 a non-leap year will probably not be necessary.
Not sure why Wiki has 8000 yr but the point is the same.