This is a bit tricky and I am not sure my answer will satisfy you.
First-order phase transitions
First, the characterizations of phase transition that you provide all correspond to first-order phase transitions. In this case, indeed, existence of a first-order transition is equivalent to the existence of a discontinuity of a derivative of the free energy with respect to a suitable parameter. (This might seem to be an empty statement, as first-order transitions are usually defined by this property, but it is not empty if you use the characterizations mentioned by the OP.)
There is already a subtlety here, because the relevant parameter,with respect to which the free energy has a discontinuous derivative might not even appear in your original Hamiltonian! The precise statement is that absence of first-order phase transition is equivalent to differentiability of the free energy (well, one uses rather the word pressure in this context) seen as a functional defined in the space of all possible (absolutely summable) interactions.
As a simple illustration, consider the Ising model in dimension $d\geq 2$ and suppose that you never thought of introducing a magnetic field term in you Hamiltonian, so that your free energy only depends on the temperature. Then, a first-order phase transition would still occur (provided the temperature is low enough), in the sense that you'll be observing distinct macroscopic phases if you repeat your experiment. But you won't see a discontinuity of a first derivative of the free energy, because you're missing the relevant parameter. Of course, in this case, this is easily repaired, but for general systems it can be a difficult problem.
Nevertheless, this "if and only if" criterion can be (and has been) used in practice to establish absence of first-order transitions in perturbative regimes (say, very high temperature and/or very low density).
In some cases, when additional nice properties are available, one can establish more workable versions that can be used non perturbatively. This is the case, for example, for the above-mentioned Ising model: the fact that the interaction is ferromagnetic allows one to prove that differentiability with respect to the magnetic field is sufficient to imply absence of first-order transitions (that is, continuity of the derivatives of the free energy with respect to any kind of "external field"). But such results are only available in very specific classes of models.
Let me also remark that, while the above criterion provides an approach to proving absence of first-order phase transitions, this is by no means the only one. There are several alternatives that establish this fact by (roughly speaking) controlling how information propagates in the system. Among the latter type of results, the most famous one is probably Dobrushin's uniqueness theorem (and the subsequent works on complete analyticity).
Continuous phase transitions
For continuous phase transitions, the possible behaviors of the free energy at the transition are so varied that it defies a precise classification. For this reason, the occurrence of a phase transition is often defined as non-analytic behavior of the free energy. Then, in each case, one can of course determine alternative signatures of the phase transition. Here, a relevant example is the Kosterlitz-Thouless transition in, say, the two-dimensional XY model. In this case, the free energy is infinitely differentiable at the transition, but not analytic. Possible signatures of the transition are the change of behavior of the 2-point correlation function (from exponential decay to power-law decay), or the change of behavior of the vortices.