I'm going to try to tackle this one piece-by-piece, because I think the questions you are asking are important and, without outside help, one can spend years extremely confused about where the basic ideas of field theory come from.
(Warning: I got a little carried away and this became much longer than I expected.)
- "Where does the idea of "field theory" come from?"
Field theories come from the very simple fact that we observe fields in nature. We see electromagnetic fields exist in response to charged particles and currents. We see velocity fields in fluid flow. We see fields describing temperature and mass density. Some of these fields (such as the fluid flow, temperature, and mass density) seem to lose their "field-like" properties at small scales (i.e. water is a collection of discrete molecules), but the electromagnetic and gravitational fields seem to be truly field-like (i.e. they are described by a continuous function everywhere in space). Thus, in order to describe nature, we need to be able to describe fields.
- "Why is field theory so important and widely applicable?"
Partly because we observe classical fields in nature, and partly because field theories that are Lorentz invariant, upon quantization, immediately give us the particles we see in nature (more on that later). However, the true power of field theory is the ability to describe any system that can be approximated by a continuous medium.
- "Is the Euler-Lagrange equation/least action principle a fundamental law of physics?"
As far as classical physics is concerned, I would definitely say yes. There isn't a (classical) theory I know of that can't be described (in some way) as a principle of least-action. If this feels ad-hoc, consider the fact that classical theories (of either particles or fields) obey equations of motion. Particles obey Newton's laws, electromagnetic fields obey the Maxwell equations, etc. These equations of motion can all be formulated in a least-action way. For many physical systems, there is more than one action that leads to the same equations of motion, so how "fundamental" a given action is is a little ambiguous, but the fact that you can derive the equations of motion for a given theory from an action principle seems to suggest that the Euler-Lagrange equations are extremely convenient and useful, if not fundamental.
Hamiltonians seem to be fundamental as well, as the Hamiltonian for a given classical path is something we can measure (namely, its energy).
- "But these explanations only explain why theories of certain physical systems turn out to be field theories, not how the field theory concept (both classical and quantum) exists in the first place."
The classical field theory concept, again, comes from the fact that we observe fields in nature, and thus we require a method of describing them.
The quantum field theory concept is a little trickier, but consider the following short thought experiment to convince yourself that we need it: An electron has been prepared in a state $|x_0\rangle+|x_1\rangle$. What is the electric field associated with this electron? If the electron were at position $x_0$, it would produce an electric field $E_{x_0}(x)$, and if the electron were at position $x_1$ it would produce an electric field $E_{x_1}(x)$. However, the electron is in a superposition of both of these positions. If you could measure the electric field, you could determine the electron's position, and so the electric field and the electron position must be entangled. In particular, the full state of the system (electron + electric field) must take the schematic form
$$|\psi\rangle=|x_0\rangle\otimes|E_{x_0}\rangle+|x_1\rangle\otimes|E_{x_1}\rangle,$$
(up to a relative phase between the two states). This suggests that, since electrons are quantum-mechanical particles and they are sources for electric fields, electric fields themselves must be quantized. The same reasoning could follow for any field the electron produces (magnetic, gravitational, etc.). This little thought experiment tells us that fields are quantized.
- "...how does one go from classical mechanics to field theory?"
In exactly the same way that one goes from classical mechanics to standard quantum mechanics! We identify a Hamiltonian and define a wavefunction. In particular, given a field variable $\phi(\textbf{x})$, you define a "wave-functional" $\Psi[\phi_0]$ which, in a sense, tells you the probability amplitude for measuring your system to have the field configuration $\phi_0$, and obeys a Schrodinger equation
$$\hat{H}\Psi[\phi]=i\hbar\frac{\partial}{\partial t}\Psi[\phi],$$
where the Hamiltonian has to be specified by your particular theory.
Usually one ditches the wave-functional approach and just defines states $|\Psi\rangle$ with $\langle\phi_0|\Psi\rangle=\Psi[\phi_0]$, where $|\phi_0\rangle$ is a state of definite field-configuration $\phi_0(\textbf{x})$. The field variable itself is then promoted to an operator $\hat{\phi}$, for which $|\phi_0\rangle$ are its eigenstates. After this, you can find the spectrum of your Hamiltonian and everyone is happy.
(Next is my own question, but one I'm sure you're asking too.)
- "So what the hell does all this have to do with particles and relativity?"
Fair question. It turns out when you perform the above procedure on electromagnetism, the spectrum of the resulting theory (QED) looks an awful lot like it contains non-interacting particles of definite momentum and spin, and whose momentum $\textbf{p}$ and energy $E$ satisfy $E^2-\textbf{p}^2c^2=0$. If the states in this theory are to describe particles, then they must be massless (since $E^2-\textbf{p}^2c^2=m^2c^4$ in relativity), and thus must travel at the speed of light. Furthermore, the energy of such a state can be shown to be $E=\hbar\omega$, where $\omega$ is the frequency of the corresponding classical plane-wave. This exactly corresponds with the results found in the photoelectric effect and the black-body spectrum. In essence, as soon as you try to quantize the electromagnetic field, photons essentially emerge (almost magically).
After you can accept that photons come from the quantization of the electromagnetic field, it shouldn't be too difficult to consider that perhaps the other particles in nature can be described by fields too. Indeed, the Dirac field (which has no classical analogue, and so is a little difficult to intuit) describes all known Fermions, the non-Abelian gauge fields (a certain generalization of the electromagnetic fields) describe gluons, $W$ Bosons, and $Z$ Bosons, and scalar fields (which are the simplest possible fields one could consider) describe the Higgs Boson.
In conclusion:
One could argue that field theory comes from the need to describe relativistic particles in quantum mechanics, and that is one way to get to the conclusion that the universe should consist of quantum fields. Another way would be to simply realize that the world consists of classical fields, and one necessarily needs a quantum description of them.
I hope this helps you understand at least the basic notion of where quantum field theory comes from and why it is so powerful.
PS: The reason you feel sketchy about the fundamental concepts of field theory is that the standard curriculum (based on books like Peskin and Schroeder) doesn't teach field theory. It more or less provides a crash course in the narrow topic of calculating scattering amplitudes from some weakly coupled field theory models. Understanding field theory goes far beyond calculating Feynman diagrams. Field theory is a much richer and broader field than textbooks would have you believe.
