It always puzzles me whenever I work on Lagrangian equations. It is easy to see that $L=T-V$ yields the correct equations of motion, but the question is, how do you get to that formula? Is it trial and error(making guesses and seeing if they actually give the correct equations), or does it have a deeper, more logical reasoning and thought behind it? Same is for all Lagrangians, from Hilbert action to the special relativistic one-how do you "derive" the lagrangian to get the correct equations. I hope you understand what I mean.