I know that defining potential for non-conservative forces is not possible and we can define potential and potential energy for conservative forces only. But can we define it for all conservative forces?
A conservative vector field is, by definition, a vector field that can be written as the gradient of a function. Since conservative forces are vector fields, they all can be written as a gradient of a function (that function is the potential)
The potential is the property of a conservative field to do work when moving a test particle. This test particle should be so small that the filed itself is (nearly) not influenced.
Test particles for gravitation are small masses, for the electrostatic filed they are small electric charges. For the magnetic field we need a magnetic monopole which does not exist or is not detected yet. But a magnetic monopole can be simulated, e.g. by a magnetic dipole made form very thin rubber, which has nearly no tension for small movements.