The gory details of this are found in the answer at https://physics.stackexchange.com/a/266046/59023.
Not the speed of sound, I want to know how lower pressure will affect the distance sound will carry and the frequency range. For example, I know that sound does exist on mars at around 600pa pressure, but travels only a very short distance. Are there any mathematical models for this?
Yes, the thing you are looking for is called acoustic impedance, which decreases with decreasing ambient pressure. You may think that a decreasing impedance would allow a sound wave to propagate further, but the reference sound intensity, $I_{o}$, depends upon the characteristic acoustic impedance, $z_{o}$, as:
$$
I_{o} = P_{o}^{2}/z_{o} \tag{0}
$$
where $P_{o}$ is a constant reference pressure here associated with the hearing threshold, i.e., ~20 $\mu$Pa at 1000 Hz (it's not flat across frequency, but adding the frequency dependence is not necessary to illustrate the main point). The characteristic acoustic impedance is defined as:
$$
z_{o} = \rho \ C_{s} \tag{1}
$$
where $\rho$ is the mass density and $C_{s}$ is the speed of sound.
The point where such a sound wave would experience strong damping is where the collisional mean free path becomes too large to support the oscillations, i.e., this would occur when the average time between collisions becomes comparable to the wave frequency. Thus, the oscillations would have no restoring force and would damp out.
In weakly damped systems, the intensity of sound decreases as $I\left( r \right) \propto r^{-2}$ while sound pressure decreases as $P\left( r \right) \propto r^{-1}$. If we look at a rough estimate of the atmospheric pressure as a function of altitude in Earth's atmosphere, we reach ~600 Pa by ~43 km.
Using the table in the answer at https://physics.stackexchange.com/a/266046/59023 for 40 km altitude, the magnitudes of $I_{o}$ and $z_{o}$ are ~1.155 x 10-10 W m-2 and ~3.462 Pa s m-1, respectively (at sea level and STP, these satisfy $z_{o}$ ~ 428 Pa s m-1 and $I_{o}$ ~ 9.346 x 10-13 W m-2).
Suppose we start with a sound intensity level of $L_{o}$ = 100 dB and we know that the intensity of the source is given as:
$$
I_{src}\left( h \right) = I_{o}\left( h \right) 10^{L_{o}/10} \tag{2}
$$
where $h$ is the altitude. So a 100 dB source at sea level would start with $I_{src}\left( 0 \ km \right)$ ~ 9.346 x 10-3 W m-2. To maintain the same intensity at ~40 km, the source intensity would have to increase to $I_{src}\left( 40 \ km \right)$ ~ 1.155 x 10+0 W m-2, i.e., increase by a factor of ~124.
The sound level intensity at a distance $r$ from the source is given by:
$$
L_{r}\left( h, r \right) = L_{i,src}\left( h \right) + 20 \ \log_{10} \left( \frac{ 1 }{ r } \right) \tag{3}
$$
where a 1 m normalizing distance is used and the source sound level intensity relative to sea level is defined as:
$$
L_{i,src}\left( h \right) = 10 \ \log_{10} \left( \frac{ I_{src}\left( 0 \ km \right) }{ I_{o}\left( h \right) } \right) \tag{4}
$$
You can see that $L_{i,src}\left( 0 \ km \right)$ = 100 dB, as we defined and so $L_{i,src}\left( 40 \ km \right)$ = 79.1 dB.
Note that sound pressure is related to sound level intensity through:
$$
L_{p}\left( r \right) = 20 \ \log_{10} \left( \frac{ P\left( r \right) }{ P_{o} } \right) \tag{5}
$$
so $L_{p}\left( r \right)$ = 100 dB corresponds to $P\left( r \right)$ = 2 Pa for $P_{o}$ ~ 20 $\mu$Pa at 1000 Hz and $L_{p}\left( r \right)$ = 79.1 dB corresponds to $P\left( r \right)$ ~ 0.18 Pa. Equation 5 shows that the sound level intensity is defined to be zero at the threshold of hearing, but if we approximate $L_{p}\left( r \right)$ ~ 0.001 dB to be the boundary of hearing then we can estimate how far away from a 100 dB one would need to be to reach this level for an atmospheric pressure of 600 Pa.
For the ~40 km altitude we used before, $L_{r}\left( 40 \ km, r \right)$ goes to ~0.001 dB at $r$ ~ 9 km (~5.6 miles). Note that 100 dB is really loud. A typical subway train arriving at a platform only generates ~90 dB of intensity. A chainsaw at ~1 m is about ~110 dB. Typical breathing is about ~10 dB. So to reduce a 100 dB sound to the equivalent of breathing in an atmospheric pressure equivalent to 40 km altitude above Earth, one would need to be ~2.8 km (~1.8 miles) away.
I wonder if the low pressure is filtering out the high frequency, or is all direct sound transmission lost and the low frequencies are being physically transmitted through the chamber floor?
The ambient pressure reaches ~5 Pa in Earth's atmosphere at an altitude of ~84 km. So if you follow the above steps and use the values in the table for 80 km found at https://physics.stackexchange.com/a/266046/59023, you should find your answer or at least a good enough approximation to figure out what can and cannot be occurring.