Solving differential equations without approximations? In physics, many problems start with a mathematical relationship of the physical phenomenon at hand, and then, in many occasion, always only leave whatever in the first order to get a nice and solvable differential equation. Then there may be terms of higher order considered later, by as far as I know, it rares goes to the third order or above.
My question is, are there any extensive analysis about how physical problems would behave when we no longer do any approximations, by keeping all orders intact?
 A: This is a quite general question. Whether or not one should use an approximation depends on several things. Disclaimer: my answer is not restricted to differential equations and contains examples from perturbation theory, but the general idea still applies.  
The most important question is whether an approximation makes sense from a physical point of view. One might lose important information when cutting of a Taylor series at low order. For example, there are cases in perturbative quantum field theory, where it is important to calculate Feynman diagrams up to several loops and in other problems, only going to tree level fully suffices in order to capture the desired physical effect. Of course, issues like convergence of the series can also play a role (to clarify: Feynman diagrams correspond to terms in a specific Taylor series, tree level is the lowest order, while higher orders are loops). 
Another question concern limitations in the: is it even possible to solve exactly? If yes, is the exact solution difficult to acquire and is it necessary? When is it necessary to use an approximation? 
There is also the other end of the spectrum: is an approximation even possible? An example would be quantum chromodynamics at low energies: due to asymptotic freedom, there is no small expansion parameter, and perturbation theory is doomed to fail.  
To summarize: the answer to your question depends on many factors, and the guiding principle should be physical intuition. What you should always keep in mind is the question "Does what I am doing make sense?".
A: If no approximations were used or all orders were kept, the outcome of the theory would presumably match perfectly the corresponding experiment that is being modeled.
The approximate equations used to model such experiments are good enough if the predictions match the results within the desired uncertainty.
As for how much some approximation is off as compared to the ideal un-approximated answer, you might find big O notation useful.
