There are distinct potentials that have the same eigenvalues but one. They are called isospectral. Supersymmetric quantum mechanics is based on this. For instance, the infinite well potential
$$
V_1(x)=\left\{\begin{array}{cc} 0 & \quad\hbox{if }\ 0\le x \le L \\ \infty &\hbox{otherwise}\, ,\end{array}\right.
$$
shares all eigenvalues with
$$
V_2(x)=\frac{\hbar^2\pi^2}{2mL^2}\left(2\csc^2(\pi x/L)-1\right)
$$
except the ground state eigenvalue $E_0^{1}=\hbar^2\pi^2/(2mL^2)$. Indeed, more generally $E_k^{1}=E_{k-1}^2$ for $k=1,2,\ldots$, i.e. the first excited state of $V_1(x)$ is the ground state of $V_2(x)$. (The eigenfunctions are quite different for the two potentials.)
I suppose this means that, unless you know for sure that you have all the eigenvalues, you cannot completely determine the potential
(This is from notes and the review of Fred Cooper, so hopefully more knowledgeable people can point to any error in my answer.)