Our physics sir made us write that gravitational potential energy is the work done in bringing a mass from infinity to a point without acceleration, but I am confused because if acceleration is $0$ it means that the external force is 0, and hence net work done should always be zero. Then how can potential energy be anything other than zero?
...if acceleration is $0$ it means that the external force is $0$...
No. If acceleration is $0$ then net force is $0$.
...and hence net work done should always be zero.
Yes, in this scenario the net work is in fact $0$, since the net force is $0$. However, this means that there are (at least) two forces acting on the object in question: gravity $F_g$ and an external force $F_e$. These two forces must be equal and opposite.
This is a standard treatment/explanation of potential energy. We move the body with a constant velocity, as $F_g=-F_e$, and so the work done by the external force $W_e$ is equal to the negative of the work done by gravity $W_g$. By definition, the work done by gravity is also equal to the negative change in potential energy $\Delta U$. Finally, if we start "at infinity" where $U(\infty)=0$ and end at position $x$, then $\Delta U=U(x)-U(\infty)=U(x)$,
Therefore, we have $$W_e=-W_g=-(-\Delta U)=\Delta U=U(x)$$ So then we have what you stated at the beginning:
gravitational potential energy is the work done in bringing a unit mass from infinity to a point without acceleration.