How can I define the Hamiltonian independent of the Lagrangian? For instance, let's assume that i have a set of field equations that cannot be integrated to an action. Is there any prescription to construct the Hamiltonian of a such system starting from the field equations?
Comments to the question (v2):
First of all, let us stress that OP is correct, that a given set of equations of motion does not necessarily have a variational/action principle, cf. e.g. this Phys.SE post and links therein.
On one hand, if there exists a Lagrangian formulation, then one may in principle obtain a Hamiltonian formulation via a (possible singular) Legendre transformation. Traditionally this is done via the Dirac-Bergmann recipe/cookbook, see e.g. Refs. 1-2.
On the other hand, if we have a (possible constrained) Hamiltonian formulation, of the type discussed in Refs. 1 and 2, then it is possible to give a Hamiltonian action formulation, which in itself can be interpreted as a Lagrangian formulation, e.g. after integration out momentum variables along the lines indicated in my Phys.SE answer here.
In other words, Lagrangian and Hamiltonian formulations traditionally go hand in hands. Thus it is unclear what precisely OP is looking for.
P.A.M. Dirac, Lectures on QM, (1964).
M. Henneaux and C. Teitelboim, Quantization of Gauge Systems, 1994.
The field equations must be conservative in a fairly precise sense in order that this can be done in a physically appropriate sense.
Then there are several Hamiltonian approaches to field theory: the De Donder-Weyl formalism and the multisymplectic formalism. Although both formalisms can accommodate Lagrangians, the can also be understood without any Lagrangian, in a purely Hamiltonian form. Both formalisms can be made fully covariant.
This holds for classical fields. How to quantize a theory in one of these formalisms is very poorly understood.