The general rule is that in a first order transition the partition function (or its logarithm, the "free entropy" or free energy over T) is continuous but its derivative with respect to the thermodynamic parameter which is driving the transition is not. The reason is that as you come into the transition from the left, you have all phase 1, and the derivative is a bulk phase 1 property. At the transition, phase 1 and phase 2 have equal bulk free energies, and phase l turns into phase 2 as you try to drive the parameter further. The partition function is continuous at the transition, because both phases have equal free energy per unit volume in bulk at this point, which is why they can freely turn one into the other.
The original idea was that at a second order transition, the derivative would be continuous and the second derivative discontinuous, allowing you to define third order transitions by the third derivative, and so on. This idea is incorrect (at least in finite dimensions, maybe it can be salvaged in mean field theory) because the second order transition is completely different from the first order in terms of the statistical degrees of freedom,
In first order transitions, the correlation functions fall off exponentially at long distances, with no fluctuations bigger than the atomic level. Nothing happens to the correlation functions of the bulk phases there, the transition is just caused by the collision of the bulk free entropy of the two phases.
At a second order transition the system at the transition point is not transitioning between two distinct phases, but has a continuum statistical limit defined by looking at the fluctuating statistical fields which are no longer "massive" (meaning they are no longer statisically clamped to a fixed value in such a way that they forget their boundary values on a large sphere). The generic behavior is that the correlations in the fluctuations of these fields falls off as a power at the transition, and this power is one of the critical exponents. Away from the transition they fall off exponentially, but at a distance called the correlation length which diverges as you approach the transition, and it diverges as another power, another critical exponent.
The properties of a second order transition is always by a statistical field theory, and these are very constrained, because they obey the analog of a central limit theorem. All properties of the transition except the type of fluctuating fields and their renormalizable "Lagrangian" are not important, or irrelevant, for the long-distance behavior. The renormalizable Lagrangians (free entropy as a function of the field, but this is the statistical analogs of quantum field theory Lagrangian because it is the weight in a path integral) have only a finite parameter space, and these parameters correspond to the number of parameters you need to tune to approach the transition. Symmetries can reduce the number of directions by forcing some parameters to be zero automatically.
The generic behavior is that the free energy as a function of some collection of bulk thermodynamic fields becomes a function of the derivatives of these fields and their values so that the continuum path integral considering these fields makes sense.