You are thinking that the "potential" takes the form of a completely arbitrary, unstructured function of the entire field configuration. Finding the minimum of this kind of function is an example of what in theoretical computer science is called "unstructured search." Indeed, quantum annealing doesn't give much of a speedup for unstructured search; it has been proven that Grover's algorithm gives the best possible quantum speedup for unstructured search, which is "only" a square-root rather than an exponential speedup.
(Actually, this isn't really correct. Finding the minimum of a completely arbitrary unstructured function is even harder than the usual decision-problem formulation of unstructured search, because it lies even outside of the search equivalent of the complexity class NP. That's because even if a quantum computer were to give you an answer, the only possible way to check it would be to effectively repeat the entire search again. So in the real world, such a quantum computer might not be very useful, because there would be no practical way to check whether its answer was correct or the computer had a bug. Also, if the search space were exponentially large, then even loading the problem statement into the computer could take longer than the age of the universe - let alone actually having the computer solve it.)
But this isn't the kind of potential that's constructed in quantum annealing. The potential isn't arbitrary but is highly structured. The prototypical problem that's attacked by QA is the Ising spin-glass problem. The details aren't important, but the point is that only a exponentially tiny fraction of all possible potentials can be formulated as an Ising spin glass. (Concretely, specifying an Ising spin glass Hamiltonian on a system of $N$ spins only requires specifying $(N \text{ choose } 2) = (1/2) N (N-1) \sim N^2$ different parameters, while a completely arbitrarily Hamiltonian requires $2^N$ parameters to specify.) Since you've specified the $\sim N^2$ different couplings but not the explicit $\sim 2^N$ different possible energies, you don't really "have an exact handle on" the potential, even though you formally know it exactly.
As an analogy, consider the Traveling Salesman problem: if I were to just list a bunch of completely trips and their lengths, and asked you to find the shortest, then this (unstructured search) isn't a very interesting problem mathematically. Clearly the only possible solution is to go through every trip one by one and keep the shortest one you've found so far. Unsurprisingly, quantum computers (of any type, not just annealers) can't give you much of a speedup on such a structureless problem. But if I instead gave you a list of distances between cities and asked you to find the shortest route that connects them all (the usual formulation of the problem), then (a) the information need just to state the (structured search) problem can be exponentially compressed, (b) there are many more math tools we can use to attack it, and (c) quantum computers might be vastly more efficient than classical computers at finding the shortest path. (I say "might" because, unlike the Ising spin glass problem, virtually no one in the theoretical CS community thinks that quantum computers will be able to efficiently solve the Travelling Salesman problem.)