As you can see from the image below, you want the variation of the action integral to be a minimum, therefore $\frac{\delta S}{\delta q}$ must be $0$. However, as you know, you might end up with another extremum.
Following the link from j.c., you can find On a General Method on Dynamics, which probably answers your question regarding Hamilton's reasoning. I haven't read it but almost surely it is worthwhile.