Is there a case where mathematical proof can replace experimentation?
Yes.
Every time you can prove that some proposition $P$ is implied by a premise $A$, and $A$ is experimentally verifiable, then you never need to experimentally verify $P$. Verifying $A$ is good enough.
As an example, Gauss' Law, $\oint E \cdot dA = \frac{Q}{\epsilon_0}$, can be proven by Coulomb's law: $E = \frac{Q \hat{r}}{4 \pi \epsilon_0 r^2}$, and vice versa. They are equivalent statements. Gauss' law is hard to verify in a lab (it's hard to measure the flux of electric field over an entire surface), but Coulomb's law is pretty easy to verify (it's easy to observe the inverse square law from a charge). Because it is proven that Coulomb's law implies Gauss' law, in principle, you need never verify Gauss' law directly; you may only ever verify Coulomb's law, and you will be exactly as confident in Gauss' law as you are in Coulomb's, because you have proof that it's implied by Coulomb's law.
Now, an objection to this example might be:
But this is just trivial. Since Gauss' law and Coulomb's law are equivalent, experimental verification of Coulomb's law is experimental verification of Gauss' law.
Okay, that's true, but without the proof of equivalence, it's not obvious at all. If we didn't have the proof that Coulomb's law implies Gauss' law, we would need to do experimental verification on both of them separately. And, because it's harder to verify Gauss' law than Coulomb's law, we would probably be less confident in the former than the latter. This is an example of a mathematical proof replacing experimental verification.
Now, my example takes a scenario where two statements are mutually implying, but this is true in general for when the implication of a proposition by a premise is one-sided, and you only need to experimentally verify the premise. Though I'm not sure if there are many examples of that.
In general, however, mathematical proofs cannot completely replace experimental validation; you always need experimental validation for any theory, and lots of it.
I just wanted to add this addendum, to justify why my answer is basically the opposite of all other, well written, and highly upvoted answers. I think it's because they are more generally trying to address a misunderstanding of the OP, which is illustrated well in this quote:
The reason I ask this is because most, if not just about all of the ToEs in theoretical physics pretty much only have their mathematics going for them. The one most infamous for this is string theory. If string theory could be mathematically proven in the way I presented, and this proof was independently replicated and stood the test of time in the same way the Pythagorean theorem has, do we need to go through all the trouble of actually making an experiment?
Okay, so let me unpack this a little. You will never mathematically prove a physical theory. You can only prove a theorem, and theorems are simply maps between propositions: if proposition $A$ is true, then proposition $B$ is true. You cannot use a proof to create a proposition out of thin air. All physical theories must start with propositions (we call them "axioms" or "postulates"). A model is constructed by starting with postulates, then mathematically proving lots of consequences of those postulates. Generally you cannot prove the postulates. If you do, then they are no longer postulates, and you needed new postulates to do so anyway. (This generally happens when we move to a more general theory whose postulates are either simpler or have more explanatory power; for example, Maxwell's equations are postulates for classical electrodynamics, but quantum electrodynamics has broader postulates from which you can derive Maxwell's equations.)
For this reason, you will always need experimental verification. And, usually, it doesn't have the clean transitive power of implication that I described above. Usually postulates are very difficult to experimentally verify, and their consequences are much easier to verify. Above, I stated that if premise $A$ proves proposition $B$ and you can experimentally verify $A$, then you don't need to verify $B.$ But quite often (especially if $A$ is a postulate), $B$ is easier to verify than $A$. But verification of $B$ does not equivalently verify $A$. Rather, failure to verify $B$, or verifying that $B$ is false does verify that $A$ is false because of the implied contrapositive. (This is how, for example, the Aether theory of light was discredited by Michelson and Morley's experiment, by showing that one of its consequences is false.)
Verifying $B$ alone doesn't necessarily verify $A$ because, there could be some other postulate, $C$ that also implies $B$. The only exception to this is if $A$ and $B$ are mutually implied, and thus, equivalent, like my Coulomb/Gauss law example. But, generally, to help build our "confidence" in some premise $A$, assuming we can't verify $A$ directly in the lab, we want to verify many of consequences of $A.$ Though we will never gain as much confidence in $A$ as we do in any of its consequences because, for every consequence $B$ of $A$, there could be some other set of premises that imply $B$. This is what makes verification of a scientific theory very difficult.
