Why should any physicist know, to some degree, experimental physics? I've been trying to design a list with reasons why a proper theoretical physicist should understand the methods and the difficulty of doing experimental physics. So far I've only thought of two points:


*

*Know how a theory can or cannot be verified;

*Be able to read papers based on experimental data;


But that's pretty much what I can think of. Don't get me wrong, I think experimental physics is very hard to work on and I'm not trying to diminish it with my ridiculously short list. I truly can't think of any other reason. Can somebody help me? 
 A: As a theorist, one likes to invent new ideas of how things might work. One crucial component to theory-building is searching the connection to experiments: A theory is physically meaningless when we cannot test it, for then it cannot be falsified. A theorist should be able to come up with experimental tests for his theories. This requires a good understanding of what experimentalists are (not) capable of. 
The perfect example here is Einstein (isn't he always?), who came up with a number of experimentally testable predictions of his theory of general relativity (those for special relativity were quite obvious, so he didn't have to work too hard on that). The most famous of these is the prediction of the correct deflection of light, confirmed by Eddington and a few others during a solar eclipse. 
A notoriously bad example in this aspect is string theory. It has thus far turned out impossible to come up with a way to test string theory, and this is regarded by many as a serious problem (although it may not have to do with the theorists' lack of understanding of experimental physics). 
A: Some cases with examples from my field (just because I know it best), but are applicable to others:


*

*Be aware of the observables. They provide starting and ending points for a theory.

*

*An example, if you are modelling 3D structure of proteins, you may be interested in generating contact maps (basically, all the pairs of atoms that are close to each other) because there is experimental evidence for them; or otherwise use them as input. You need to know the limitations of this data to see if the differences between your theory and the experiment are significant. You also need to be aware of the differences between, for example, plants and humans at a protein level, what works fine in one may not do so for the other.


*Some parts of your theory may be very difficult to describe analytically. But one can, for example, try to just apply Machine Learning to the data and just use it. This means that you need to be aware of its limitations, as well as what aspects may be improved in a feasible time span (slight technological improvements, different settings, etc).

*

*A flux of electrons hitting a protein will tend to fragment it, and some spots are weaker than others. This may be solvable theoretically by some Monte Carlo simulations and perturbation theory, but in practice is undoable (or at least, it has been so far). But there are thousands of machines that are generating literally gigabytes of data every hour. One could just gather enough of this to get a good model. And not a single wavefunction was needed.

*In the previous apparatus, one can get less noise and better resolution just by increasing the integration time. This means that we get less samples, but they are going to be more accurate. If your particular application would benefit more accuracy, you know that  you can obtain a few folds with today's machines. On the other hand, other sources of noise cannot be easily improved, and one has to find the way to deal with them.



I want to add that in the second case, depending on the ML algorithm used, one can actually make a physical interpretation of the parameters. Something similar to the liquid drop model for the nuclear masses: it is just a fit of many parameters, but interestingly, for some of them that can be (at least partially) modelled theoretically, the values are in the same range. 
A: Because otherwise you are a mathematician. 
The point of Physics is to describe the nature using the language of maths, but the only ways to stay in contact with nature is to interact with it through experiments and observations.
If you completely lose the ability to grasp how a process starts and develops, how much it can be influenced by external factors, how to extract significant data in order to understand it and reproduce it; then you are just playing with numbers. You may find interesting things but you are not doing Physics any more.
Moreover nowadays lots of theories get tested with computational simulations which share many of the techniques being known for ages by the experimentalists, especially in data analysis. Getting your hands dirty from time to time, will make you a much better physicist, not only when it comes to designing an experimental test for your work: indeed this would be an easy task if you had kept your model close enough to nature.
A: Here's a reason that hasn't been touched yet (but is alluded to by your question): to be able to form new theories.
A lot of the most interesting theories in physics comes from someone reading about an experiment and trying to explain the results. We wouldn't have relativity if Einstein didn't read about the Michelson–Morley experiment and going "hmm.. let's assume there are no errors, something funny is going on here".
There's still a lot of experiments published with unexpected results with incomplete or not so convincing explanations. Yes, a lot of them are in less glamorous fields such as fluid-mechanics or acoustics or crowd-dynamics. But once in a while we get interesting theories out of them and once in a while two seemingly unrelated fields yield a single unifying theory.
A: For me, an experimentalist, the number of theoretically inclined people  I have observed here, who are floundering with concepts that should be philosophy and who navel gaze about collapse of wavefunction,  amazes me.
I would order a course in particle physics, this will give an intuition of what it means to move in the quantum mechanical dimensions, a connection with reality and hard numbers.  Without a clear map of the real numbers we have mastered that describe nature a theorist is just a mathematician, as far as intuition goes. That is why we get people who think they have found "compositness", or a new way of seeing nature: because they are ignorant of the bulk of hard data which has been built up over the years and has to be incorporated in any higher order theory.
In this sense, the strong support of string theories by a lot of physicists comes because it has the group and equation structure to   embed all the hard won  measurements  of the past decades in a coherent framework.  On the other hand maybe this is what has made theorist think that they can just cogitate and create physics theories, because string theory is a theory validated by theories that have been validated by data. Lets hope that more predictions than supersymmetry, though if it is found it will be great enough, and large extra dimension models will be offered for testing in the next generation of measurements at LHC and possibly the ILC.
A: I don't know if it helps, but perhaps breaking down what can be verified into what are the measurable quantities might be of some help? Perhaps also within what bounds the quantities from the model are valid. 
I must admit as a failed physicist (I think at all levels) to me theoretical physics is perhaps is more applied mathematics in the sense that mathematical concepts are applied to tackle physical problems whereas mathematicians are more concerned with developing mathematical concepts. Of course, that doesn't mean that some concepts start life in the hands of theoretical physicists and are taken over by mathematicians and investigated in more depth. Both groups are necessary and important. 
I must admit that to me the navel-gazing (which I think is a little harsh) such as kicking the tyres on QM (measurement problem etc) are essential to really understand the limits of the model. It's only by trying to go beyond the current limits of our understanding that we progress.
For my part I must admit that I still have reservations about perturbation theory (to me it still seems like trying to fit a square peg into a round hole by shaving bits off - it fits, but is it right). But, that could possibly be because I understand very little.
