First question is what set of first principles would one expect to derive the Navier Stokes equations?
This question is Simple. In physics, a calculation is said to be from first principles, or ab initio, if it starts directly at the level of established laws of physics and does not make assumptions such as empirical model and fitting parameters.
And the second, and main question is why does a derivation fail?
The main Problem in Turbulence is it's scaling. We need to use empirical factors to correct the results if the dimensions of the system is changed. Due to the complexity of 3D flow equations, this key problem has lead to a situation where practically all problem solving efforts are trying to mathematically found the link between Measured data and the equations themselves.
This must fail because the current equations are WRONG. (Feynman, Key to Science)
This comes from the simple fact, that it disagrees with the experiment.
Are we missing some yet to be discovered set of first principles in this area of physics?
This obviously must be the case. And as the equations clearly can't be made any more simple as they already are, they must be too simple at the moment. Some aspect must be missing.
I have personally made an invention, which I have patented and also tested in full scale in a Lab. This invention was based on my idea about the cause of Turbulence; and I indeed really managed to make the [noisiest turbomachine ever]5 to a high efficiency and vibration free smoothly running machine.
I really killed the turbulence. This caused us to destroy a 3-hole pitot pipe in the Lab, due to a laminar-alike fluctuating flow. The flow just wasn't turbulent, as you would expect it to be.
The solution was based on the idea, that I need to prevent the fluid to "break on pieces" because of sudden shock. This meant in my thoughts that the viscous forces can't be transferred through the Fluid as there is intern surfaces, which can only interact with collision and friction.
Now, if you Look Navier Stokes Equations, you immediately notice that viscosity is just not handled that way. Though this idea is pretty simple, and I immediately found some predicting success from it. It’s just a mathematical horror to try add these aspects on the 3D Navier stokes equations. First we need a scale free limit and definition, which tells us when exactly we should calculate viscosity, and when friction and collision.
Imagine a Kinetic gas theory, where you would have a particle velocity depended particle size?
But I actually found the way, and I was able to derive these modified Navier Stokes Equations in such a matter, that this "mess" can be handled statistically like in Kinetic gas theory. After I got the Idea of the path, it was just straight forward; The equations were relatively simple, and the mathematical results fitted perfectly to old Measurement data.
Just to verify that this works, I have also succesfully calculated few turbulent Pipeflow losses with this new model.
Yes. The Navier-Stokes are incomplete. Universal Continuous and Smooth solutions does not exist, as the aspect that fluid breaks in pieces due to the accelerating conditions which can be defined with Froude number $\sqrt3$. The energy dissipation of this breakdown can be treated statistically, and this provides perfect match to experimental data on all scales.