You can use the formula
$work = force \times distance$
to imagine it intuitively.
When the pushing force is close to the pivot, the force moves through a small distance (for a given angle of rotation), the work done is low, so the kinetic energy gained by the door, related to the speed, is low.
When pushing with the same force further from the pivot, the distance is larger, for the same angle, so the kinetic energy gained and speed of the door would be larger.