You must think displacement as the shortest distance between two points. Distance is the actual distance you travel while going between two points.
In the above example, the length of the black line gives the displacement between the two points $A$ and $B$ and the length of the brown line between the two points $A$ and $B$ gives the distance traveled.
By definition, velocity is given by:
$$\vec{v} = \frac{\Delta(displacement)}{\Delta t} = \frac{d(displacement)}{dt}$$
By definition, speed is given by:
$$s = \frac{\Delta distance}{\Delta time}$$
In your case, you return to the same point, therefore, you have a net displacement of zero but the net distance traveled is not zero.
As the net displacement is zero, your net velocity is also zero. As your net distance is not zero, your average speed is not zero.
Useful references:
distance and displacement of an object on a straight linedistance and displacement of an object on a straight line
What is displacement? Position relative to a reference point or change of positionWhat is displacement? Position relative to a reference point or change of position
