Your question requires an answer on three different levels:
- mathematical;
- pure thermodynamics;
- statistical mechanics.
1. mathematics
The definition of the Helmholtz free energy you refer to is nothing but the Legendre transform of the fundamental equation $U(S,V,N)$ with respect to its first variable $S$ in term of the conjugate variable $T=\left( \frac{ \partial{U}}{\partial{S}} \right)_{V,N}$. The original Legendre transform would be easily done by requiring $U$ to be a twice differentiable function of $S$ with a positively defined hessian matrix. However, such a request is too strong for real thermodynamic systems. It is well known that a useful extension of the Legendre transform is the so-called Legendre-Fenchel transform (LF) (or convex-conjugate).
In the case of thermodynamics, the definition of LF transform is slightly different from the most common mathematical definition. In the case of the Helmholtz free energy, it would be written as
$$
F(T,V,N) = \inf_{S}( U(S,V,N) - TS )
$$
Such a definition reduces to the usual Legendre transform in the part of the domain of $U(S,V,N)$ where the function is strictly convex (and twice differentiable with respect to $S$). Where the function is convex but not strictly convex (i.e. what mathematicians call an affine function, i.e. a linear function), the LF transform maps the entire affine interval into a single point where left and right derivatives differ.
Since fundamental equations must be convex (o cancave) but not strictly convex (or strictly concave), it turns out that the LF transform is the proper mathematical tool for a change of variable $S \leftrightarrow T$.
2. Pure thermodynamics
Affine regions of $U(S,V,N)$ are to be expected, due to the phenomenon of phase coexistence. In such regions, the thermodynamic potential must be a linear function of its extensive variables since it corresponds to an equilibrium condition of an inhomogeneous system made by more coexisting phases. At coexistence, $T(S,V,N)$ is a constant as a function of $S$. But this is physically consistent with the presence of a latent heat at the first order phase transition.
3. Statistical mechanics
Statistial mechanics is deemed to give access to Thermodynamics, by starting with a model for the Hamiltonian of the system. However, such a program in general requires the so-called thermodynamic limit (TL). TL is required for different reasons. Summarizing, these are:
- only TL can introduce the non-analiticity required to recover phase transitions;
- only at TL (if it exists) it is possible to recover the extensiveness
- only at TL (if it exists) it is possible to recover the convexity properties.
Without TL many properties, which are considered typical for thermodynamic systems, would not be valid. On the other hand, working with a finite number of degrees of freedom, although unavoidable from the numerical point of view, in general introduces non-convex (unphysical) regions. Therefore, TL is required, but at TL the $T(S,V,N)$ is not invertible for $S$ in the whole coexistence region. Nevertheless, LF transform can cope without problem with the situation.