In this first part of the answer, I'll discuss the possible generalizations of the relativistic quantum particle based on the Poincaré group (to systems which can also be considered as quantum particles).
Then, in the second part, I'll discuss their classical counterparts in terms of coadjoint orbits. It seems that you already know this result as you stated that in the question, so I'll try to give you some heuristic motivations of this correspondence.
The definition of a relativistic quantum elementary particle, according to Wigner, (and as explained in the attached answer), is a quantum system carrying an irreducible unitary representation of the Poincaré group. Also as you mentioned, in the non-relativistic case, the Poincaré group is replaced by the Galilean group.
The Poincaré and Galilean groups are special cases of what is known as kinematical groups. A kinematical group is a group of automorphisms of space time. Bacry and Lévy-Leblond have classified the possible kinematical groups for 4-dimensional space times and found 11-possible kinematical groups. Their results is briefly described in the introductory section of the following work by Alan Mcrae. For completeness, I'll quote Bacry and Lévy-Leblond's general conditions defining a kinematical group:
(i) Space is isotropic and spacetime is homogeneous,
(ii) Parity and time-reversal are automorphisms of the kinematical group, and
(iii) The one-dimensional subgroups generated by the boosts are non-compact.
The 11 possible geometries include the de Sitter and anti-de Sitter as well as other geometries known from cosmology and other applications in quantum field theory.
Sanjuan found in 4-dimensions a one to one correspondence between the Bacry and Lévy-Leblond classification and the Cayley-Klein geometries.
Thus in summary of the first part, a general quantum particle can be taken as a quantum system carrying an irreducible representation of one of the above kinematical groups.
Souriau defined a classical elementary quantum system as a homogeneous symplectic manifold. In the case of particle, it will be a homogeneous symplectic manifold of the kinematical group. It is a theorem by Kirillov that a homogeneous symplectic manifold is locally isomorphic to a coadjoint orbit (which is almost a coadjoint orbit), please see Kirillov (section 1.5) for explanation of this point.
The reasoning of Souriau's definition, is that the classical elementary system should be a classical system that when quantized, should reproduce the irreducible representation defining the quantum particle. It is known that coadjoint orbits have such a property (in the non-compact case, only for certain representations).
More deeply, Souriau's definition stems from the fact that irreducibility and unitarity in the quantum version correspond to transitivity and symplecticity and in the classical version.
The manifold representing the particle needs to be symplectic rather than Poisson, since on a symplectic manifold the values of the Casimirs are constant and the Casimirs correspond to physical properties of the particle such as mass and spin.
The correspondence between irreducibility and transitivity is intuitive. However, it relies on deep theorems in induced representations or alternatively in $C^*$ algebras. I think that your supervisor Prof. Landsman is a world-class expert in these matters. He can surely provide you with a better answer. However, I'll try to give you two heuristic explanations.
First, the representations associated with the coadjoint orbits can be realized as coherent state representations, i.e., spanned by coherent vectors. In the semiclassical limit a coherent vector becomes highly concentrated at a phase space point, thus if the representation is irreducible it should be able to transform between any two points in the coadjoint orbit.
The second explanation is that the group action on the coadjoint orbit is Hamiltonian, i.e., there exist functions on the coadjoint orbit whose Poisson brackets realize the algebra of $G$. The momentum map heuristically is a matrix in the representation space whose appropriate elements are the Hamiltonian functions. The group acts on this matrix by a similarity transformation. This matrix also provides a parametrization of the coadjoint orbit thus if the action is reducible, then consequently there exist regions on the coadjoint orbit closed under the group action.
To summarize the second part, the classical models corresponding to the quantum particles are the coadjoint orbits of the kinematical groups.
Remarks:
- The kinematical groups are non-compact and in this case, the coadjoint orbits do not correspond to all the representations, for example there are difficulties to associate a coadjoint orbit to the complementary series. Some of the kinematical groups possess complementary series; I don't know what is the physical interpretation of their complementary series.
- Souriau went a further step and defined a classical scattering operator between symplectic manifolds, (in our case, the coadjoint orbits associated with the particles) called symplectic scattering. Please see an application in this article by Duval Horváth and Horváthy.
- Geometric or perhaps other quantization methods (e.g. Berezin quantization) of the coadjoint orbit should produce the irreducible representation as the quantum space. In the reverse direction, dequantization, should produce the coadjoint orbit by means of the action of the group on the Highest weight.
