I'm not sure i'll be able to post all the links i'd like to (not enough 'reputation points' yet), but i'll try to point to the major refs i know.
Matilde Marcolli has a nice paper entitled "Number Theory in Physics" explaining the several places in Physics where Number Theory shows up.
[Tangentially, there's a paper by Christopher Deninger entitled "Some analogies between number theory and dynamical systems on foliated spaces" that may open some windows in this theme: after all, Local Systems are in the basis of much of modern Physics (bundle formulations, etc).]
There's a website called "Number Theory and Physics Archive" that contains a vast collection of links to works in this interface.
Sir Michael Atiyah just gave a talk (last week) at the Simons Center Inaugural Conference, talking about the recent interplay between Physics and Math. And he capped his talk speculating about the connection between Quantum Gravity and the Riemann Hypothesis. He was supposed to give a talk at the IAS on this last topic, but it was canceled.
To finish it off, let me bring the Langlands Duality to the table: it's related to Modular Forms and, a such, Number Theory. (Cavalier version: Think of the QFT Path Integral as having a Möbius symmetry with respect to the coupling constants in the Lagrangian.)
With that out of the way, I think the better angle to see the connection between Number Theory and Physics is to think about the physics problem in a different way: think of the critical points in the Potential and what they mean in Phase Space (Hamiltonian and/or Geodesic flow: Jacobi converted one into another; think of Jacobi fields in Differential Geometry), think about how this plays out in QFT, think about Moduli Spaces and its connection to the above. This is sort of how I view this framework... ;-)