I'm confused by the famous wave-particle duality mystery:
When a particle is left unobserved, it acts like a wave and can explore all classically available particle trajectories simultaneously. By looking at it, you force it to decide on a single trajectory, like going through the left or right slit, or like Schrödinger's cat that ends up being either dead or alive; the wave-like characteristics are lost.
Are there theories that actually explain this behaviour?
The answer by JKL is sufficient but I want to address particularly the Why?.
I am so confused. Why does it act the way it does?
If one reads a bit about the history of science and physics in particular, it becomes clear that physics at the ultimate end does not answer the ultimate Why?. Physics posits laws and uses sophisticated mathematical tools to theorize from "axioms", get equations, and check against the data experimentally. It finds How, from "axioms" one ends with predictions for measurements that validate them.
The Why? questions addressed to the "axioms" has the only answer: Because .
When one is validating a theory, as for example Newton's gravitational theory, and one hits a disagreement with the data, new "axioms" and new theoretical tools are developed to explain the Why of the disagreement and the new theory validated for the regime of disagreement. Special and General Relativity are an example. The history of physics has other examples : Thermodynamics, developed theoretically to explain bulk behavior, statistical mechanics, out of classical mechanics emerge out of asking How and assuming "axioms" to contain the Why. Each theory with its own regime of validity.
Within a theory a question with a Why is answered by proofs of How finally hitting on the "axioms". Quantum mechanics is the last in the series of exploring the microcosm. The Why you are asking hits against the Because of its "axioms".
There are people who continue the exploration, trying new axiomatic theories of how a quantum mechanical theory can emerge from a smaller regime where we are back to classical concepts, and contain the Why in "axioms" for their new theory. They are not successful except with small models. The bulk of theoretical physicists either ignores their efforts or proves that their new theories would violate a basic validated law as, for example, Lorentz invariance. And that is the story of Why in Physics. Ultimately the answer is Because.
Edit: "axioms" in quotation marks , to include mathematial axioms and physics postulates. Physical theories start from strict axiomatic mathematical theories where everything is contained and rigorously self consistent and add postulates in order to connect physical observables to the mathematical variables/functions. Postulates are a necessary choice to turn a mathematical theory into a physics theory and in classical theories are referred as "laws". I call them "axioms" because they are the foundation stones of the theories, if one goes the whole theory goes.
This answer expands on the answer given by anna v
It is easy to become accustomed to a pattern of physics providing explanation after explanation.
For example, how is it that a gas has elastic properties under compression? If you compress it to half the volume, the pressure pretty much doubles. It's surprisingly similar to a spring, yet a gas isn't a spring, surely?
Physics offers a wonderful explanation: the molecules of the gas are moving fast, and collisions of molecules are perfectly elastic. The pressure of a gas against the walls of the vessel arises from elastic collisions of the molecules with the walls. If you half the volume the number of collisions pretty much doubles, hence the change of pressure.
What the physics did there: it moved the description of physics taking place to a deeper level. As it happens the physics on this deeper level (fast moving, elastically colliding molecules) is very readily visualized in things we know in the macroscopic world. We can readily visualize a population of bouncing balls.
In this particular case we are fortunate: the description at a deeper level consists of something that has a counterpart in physics we can see happening with our own eyes.
So yeah, it's natural to hope that a theory at quantum level can describe the physics taking place in terms of things we can visualize: particles, and waves. Well, in the case of quantum mechanics it turns out that what is expressed by the equations has no counterpart in the macroscopic world.
We make do with concepts we can visualize; particles and waves. In that sense the duality is introduced by the human thinking process; it's not inherent in the physics.
When newtonian mechanics was established it looked for more than 300 years that no deeper level existed. Then, in the course of development of relativistic physics, theory of motion moved to a yet deeper level. So we learned not to expect that we are at the most fundamental level. A deeper level of description may be possible, we don't know.
Physics is not in the business of looking for exhaustive explanation. For the duration of working at a particular level of description (which can last hundreds of years) the assumptions necessary for that level of description are taken as given.
For comparison: Newton assumed the existence of a gravitational force, acting over large distance, to explain the fact that the planets are orbiting the Sun. Newton's contemporaries were more demanding. For example, Descartes offered a hypothesis of vortices in space, with those vortices exerting an inward push on the planets. Descartes was determined to restrict himself to hypotheses with understandable elements only. He could visualize vortices, thus he ended up with vortices.
In retrospect we see that Descartes was just bogging himself down. Newton was more successfull because he operated on a need-to-assume basis.
Newton could not explain gravity itself, but he saw the strong explanatory strength it offered, and he moved ahead.
Are there theories that explain wave-particle duality?
Yes, several actually, but they are interpretations rather than new theories.
As anna v explained in her reply, at its most fundamental level the only answer a physical theory can give to the question Why? is Because!.
It's impossible to give a more meaningful definite answer until we've taken another step down the rabbit hole (assuming we've not yet arrived at the bottom, of course).
However, you're not the only one uncomfortable with the situation, and that's where interpretations of our physical models come in, and there's a whole bunch of them for quantum mechanics.
As far as wave-particle duality goes, here are the explanations given by various interpretations:
The Copenhagen interpretation is missing from this list by intention ;)
You will probably find a satisfactory answer reading the experiment done by Shahriar Afshar, who claimes to have observed both, the wave and particle, properties of mater simultaneously. Here is the link,
http://en.wikipedia.org/wiki/Afshar_experiment
You will find more links therein, as well as criticism of the interpretation of his experiment. The best way to settle scientific "conflicts" is by experiment. Debates and arguments have conceptual limitations.
I think your confusion about waves and particles will go away if you start thinking about what information nature allows you to know about the system you are examining.
Think about macroscopic ball: billions of air molecules and photons bounce off from it all the time. You can in theory use all the information to have lots of data about the ball: it's position shape, color, etc. It's all behave classical because of the so many clues the nature give you.
But things become interesting if you think about a single particle. You cannot see a photon flying along the same way you see a ball flying after it was kicked. The only way to observe a photon is catching it in a detector.
Imagine we excite a single atom, that electron will soon drop back to a lower orbital and the will emit the photon. What do you know about this photon? Once it's emitted we can be pretty sure that after 1 nanosecond it will be roughly 30cm away, after 2 nanoseconds it will be roughly 60cm away as it's moving precisely at the speed of light. So if it didn't hit the wall 40 cm away from the emission site within 2 nanoseconds, we can be certain it never will.
We also don't know which direction it was emitted. Any direction possible. If we plot the possible paths the photon may take as a probability density function and animate it as the time passes it really start looking like a spherical wave going away from the emission site.
Then we don't even know when the photon is emitted, we can know a probability density function when the emission happens after excitation. So our expanding wave like sphere now really looks like a pulse of wave not like a sharp sphere. Then we can add further twists by adding optics to the system, mirrors will reflect the photon, if we address this in our calculations it looks like the wave is reflected from the mirror, or refracted on the prism like real waves do.
Then the final twist is when we deal with the case when the photon can reach the the same point in space via multiple paths at the same time: Feynman's path integral formula can be used to calculate what's the chance we can catch it there. That's why the interference happens.
All of these can be derived from the information what we can know about the photon, it doesn't mean there is a real wave there. It just exist our heads, while trying to reason about the information we know about the photon.
Eventually one atom somewhere will absorb the photon and nature will finally reveal where the photon actually went and you can verify that it behaved according to the laws of Physics, but until that, you can only guess, and that's why it seems there are waves.
