Boundary layer theory come in as a method to "simplify" the mathematics in fluid mechanics, so that it is solvable analytically. It separates the fluid as two regions:
- where the viscosity effect is important, i.e. boundary layer
- where viscosity is not important
This approach has been proven useful for large range of applications, that's why aerodynamics (for example) has flourished and matured a lot in the past century. In some cases where Navier-Stokes has an exact solution, researcher had shown that there exist a boundary layer where viscosity is important, and outer region where it is negligible. These examples are abundant in Fluid Mechanics book, such as Kundu-Cohen's, or Landau-Lifshitz's. By the nature of this theory, it would be interesting if there are rigorous expositions mathematically as you may probably want to see. However, if Landau's treatment did not satisfy you, you may want to check a book by Oleinik and Samokhin titled Mathematical Models in Boundary Layer Theory.
A probably important thing to note, boundary layer approximation may cause erroneous result due to the assumption of the thin boundary layer is not satisfied. Such an example (quite recent) can be seen in an article by Doinikov and Bouakaz in JASA: