(...) Depending on what type of physics you are working on the mathematical models change (...)
I don't understand the meaning of this sentence, because I always thought that, in general, we have to represent the physical system in some mathematical form like $\mathbb{R^n}$, the space of the functions on $\mathbb{R^n}$, or others, and then figure out which properties this mathematical space has.
Physics is essentially about describing nature using maths.
From an abstract point of view, this means to define some properties of interest in whatever physical system you are studying, attach numbers to these properties, and find a way to describe/predict how these properties interact/evolve.
This process leaves you with a bunch of numbers. You can always reduce to the situation in which these are real numbers${}^{\textbf{(1)}}$, and therefore you can think the set of properties as points living in $\mathbb R^n$ for some $n$.
Note that, in general, it won't be useful to add a vector space structure to $\mathbb R^n$ (for example when some parameters describe some finite degrees of freedom, say a "colour" or something like that), although this is often the case.
In this sense, yes, you can always represent a physical system as some point in some Euclidean space $\mathbb R^n$, and then study how this point evolves in time, or more generally the properties of the subset of $\mathbb R^n$ that describes your system in this particular mapping.
Note that there isn't a lot of information in this statement: it's essentially just a mathematical reformulation of what it means to describe a physical system using maths.
The problem is that in quantum mechanics it looks like we can write $|\alpha\rangle$ meaning the real state in the real world, not the mathematical object that represents it, and we say that the space of the states in the real world is a vector space, not the space that represents it.
I feel like this is mostly a matter of terminology. A "quantum state $\lvert\alpha\rangle$" is nothing but a vector in a complex space (or more precisely, an element of complex projective space). Again, you can think of this as a point in some Euclidean space $\mathbb R^n$ (such a description would be non-ideal from a technical point of view, but that's not really important here).
The set of numbers you use to describe a quantum state is your way of describing the physical system. There is no difference between this and what you have in the classical case from this point of view.
Sure, it's much harder (if possible at all) to get an intuitive understanding of how quantum mechanics works at a fundamental level, but that doesn't mean anything here: the numbers you use to describe a system are still just parameters that make sense within the way you are describing the system, they are not "the real state" themselves.
So this confuses me because a vector space needs a sum operation, but I feel like you can't say what is sum without a mathematical representation.
I'm not totally sure where you are getting at with this. From a fully general point of view, you do not always want to deal with a vector space structure, though that is an extremely common and useful case. The "sum" operation in the set of properties you are studying will make sense when there is such a vector space structure in your description of the system.
So I'm asking you if in physics I have to say that a physical system is itself a certain mathematical structure or I have to say this physical model can be represented with this mathematical space that has this structure.
I would argue that "this physical model can be represented with this mathematical space that has this structure" is a more precise way of putting it, but physicists will often not bother with distinguishing between the two statements when discussing about physics, as in practice the distinction is inconsequential from a practical point of view.
${}^{\textbf{(1)}}$ If the properties can be listed using integer numbers, these are still subsets of $\mathbb R$. Similarly, complex numbers, operators, or matrices, can always be thought of as a bunch of real parameters.
