As you can see from the image below, you want the variation of the action integral to be a minimum, therefore $\frac{\delta S}{\delta q}$$\displaystyle \frac{\delta S}{\delta q}$ must be $0$. Otherwise, you are not taking the true path between $q_{t_{1}}$ and $q_{t_{2}}$ but a slightly longer path. However, even following $\delta S=0$, as you know, you might end up with another extremum.
Following the link from j.c., you can find On a General Method on Dynamics, which probably answers your question regarding Hamilton's reasoning. I haven't read it but almost surely it is worthwhile.