This borrowing energy for a time is not a very helpful way of thinking. It was required for old-fashioned perturbation theory, which was superseded by QED in 1948.
Both theories had diagrams, but the former were time-ordered and did not conserver energy at the vertex.
The Feynman diagram explicitly conserves energy and momentum at the vertex, so using this principle in the context of a Feynman diagram is just wrong.
Consider:
$$ p \rightarrow n + \pi^+ \rightarrow p $$
Obviously a violation of kinematics (btw: Feynman diagrams are Lorentz invariant, and not in position space, so specifying that the proton is "at rest" is meaningless--but it may be useful to work out some invariants since the math is easier).
So the proton 4-momentum is (in it's rest-frame):
$$ p_{\mu} = (M_p, 0,0,0) $$
so the intermediate state has to sum to that
$$ p^n_{\mu} = (E_n, \vec p) $$
$$ p^{\pi} = (E_{\pi}, -\vec p) $$
with:
$$ E_n + E_{\pi} = M_p $$
There are an infinite number of $\vec p$ that solve this, provided:
$$ E_n \ne \sqrt{M_n^2 + p^2} $$
$$ E_{\pi} \ne \sqrt{m_{\pi}^2 + p^2} $$
So they're off-shell. That's it.
At the end of the day, they're just terms in an approximation, not real particles. That they can be extremely useful for understanding your interaction (e.g., Rosenbluth separation) is why we treat them with so much respect.
Edit: Don't give up on the time-energy relation, though. It's very useful (see: Breit-Wigner distribution), or the concept of the "width" of a particle (or spectral line).