Let's take an CIE XYZ color space as an example. There are many colors that are outside our gamut - how do we see such a color?
You will see a slightly different colour inside your colour gamut if this is a visible colour, or nothing if it is not.
Consider a monochromatic 300nm light; this is below our 380/400nm lower frequency limit, so it will be invisible. A bichromatic light consisting of 50% 300nm light and 50% 700nm light will just look like a dim red light, as the 700nm light is being absorbed by the red cones in your eye.
All you have to do is calculate the dot product of the colour's spectrum with those of the red, blue and green cone response curves, to get 3 values for the red, green and blue. Normalize and you have a colour index that you can compare to other colours.
You can only see colours inside your colour gamut, by definition. So everything else is either invisible, or turned into something inside that gamut.
There are some interesting effects in low light, when your rods are active, but the only colour you can mix with that greyscale vision is red as rods are insensitive to it. A small proportion of the female population supposedly are tetrachromats, as they have different cone genes on each X chromosome, giving them four types of cone instead of three. Though their gamut would be larger, and have an extra dimension, they too would only see what is inside their gamut.
Possibly with afterimages. Near where I live there's a broad bunch of vividly magenta flowers along the sidewalk. I stare at them for a minute, then look at the surrounding green trees, bushes and grass. The kinds of green I perceive are probably not reproducible by any physical means without afterimage effects taking place.
You start by talking about the xyz space. So I assume that you are wondering what kind of colors lie outside of that typical 'horse-shoe' shape.
As Phil H. explains, the XYZ shape is indeed defined by three color response curves (inspired by the human vision). So, the XYZ values are already 'confined' within the limitations of the human visual system.
So, where does that horse-shoe come from then?
Notice how these response curves overlap. In other words: Any wavelength that triggers the green receptors, will also trigger to some extent the blue or red receptors. So, not every XYZ combination can be achieved with a color spectrum. It's this overlap that defines the horse-shoe shape.
Inside the horse-shoe, you have XYZ values for which there exists at least one color spectrum. Outside of that horse-shoe, are XYZ values that could only be reproduced with a spectrum that is (partially) negative: physically impossible colors.