30 cm diameter implies a 100 cm track length (roughly - less as you move further in) traversed in about 2 seconds - or 50 cm / second. The radius of the needle is specified in the standard as less than 25 $\mu$m. As the frequency increases, the ability of the stylus to follow the groove is limited. 10 kHz would have a "wave length" in the groove of 50 $\mu$m, which is starting to be at the limit!as, as that is the diameter of the stylus. (but see below for a more detailed analysis)
The smaller the amplitude, the higher the frequency you can record; and the same is true for the stylus. There need not be a hard limit: but existing equipment was optimized to work with the audio range of today's humans so expect to have to design a new stylus, amplifier, equalizer, ... If you want to extend it.
UPDATE
I looked up some dimensions of the vinyl record groove to put a bit more "physics" in my answer. I found the RIAA dimensional standard for an LP record from which I reproduce the following:
desirable tip radius 0.5 mils = 12.5 µm
outermost groove diameter 11.5" = 292 mm
groove width minimum 0.0010" = 25.4 µm
innermost groove diameter 4.75" = 120 mm
The "Handbook for Sound Engineers" (ed.Glen Ballou) also has a wealth of information. For example, I found the following diagram on page 1146:
This nicely demonstrates what I was describing earlier - that when the modulation of the groove becomes small compared to the size of the stylus, you will have trouble reading the signal faithfully. But where my earlier calculation used the "no bigger than" specification, this diagram uses the "typical values for high end stereo stylus" - and there's a difference.
Note that the amplitude of the signal is given in "cm/s lateral". The reason that this is a sensible unit is understood when you think about the mechanism of a cartridge: a moving magnet induces a signal in a coil, and the voltage induced is a function of the rate of change - that is, the velocity of the stylus. This means that the physical amplitude of the signal would decrease with frequency for the same signal strength. In reality, in order to allow for a thinner groove, the RIAA equalization will boost the size of the high frequency signal at recording time - in effect increasing the amplitude of the high frequency signal. During playback, the opposite filter is applied. This reduces the amplitude of the high frequency signal - and incidentally takes out the high frequency noise from the surface of the vinyl at the same time:
Now the frequency at which the stylus becomes unable to track the groove faithfully can be calculated - this happens when the curvature of the groove is smaller than the curvature of the stylus. If we have a sine wave of physical amplitude $A$, that is $y = A\cos(x/\lambda)$ then the velocity of lateral displacement $v = \frac{dy}{dt} = \frac{dx}{dt} \frac{dy}{dx} = v_t \frac{dy}{dt}$ where $v_t$ is the track velocity.
It follows that the lateral velocity $v = -\frac{A}{\lambda}\sin(x/\lambda) v_t$. For a given velocity amplitude $v$, we find that $A = \lambda \frac{v}{v_t}$. Now at the peak, the curvature is calculated from the second derivative of the displacement, $$\frac{1}{|R|} = \frac{A}{\lambda^2}\cos(x/\lambda) = \frac{1}{\lambda}\frac{v}{v_t}\cos(x/\lambda)$$
Evaluating this at the peak, where the $\cos=1$, and writing the wavelength in terms of the frequency and track velocity, $\lambda = \frac{v_t}{f}$, we get
$$|R| = \frac{v_t^2}{vf}$$
This is interesting: as we move from the outer rim of the record (246 mm diameter) to the inner rim (120 mm diameter) the track velocity is cut by 2.4x; this means that if the amplitude of recording is unchanged, the frequency range is cut by almost 6x! Also, as the amplitude of the signal is doubled, the frequency that can be recorded without distortion is halved. So while the record might have a "flat" response, the "distortion curve" does not.
Let's calculate the limiting frequency (where the stylus no longer tracks the signal) at the innermost groove of the record. We put $R = \rm{4~µm}$ (for a "high quality stereo stylus" per the Sound Engineer's handbook), $v_t = \rm{121~mm * \pi \frac{33}{60} = 209~mm/s}$, and $v=\rm{3.54~cm/s}$ (again, per SEH). We find
$$f = \frac{v_t^2}{vR} = 300 kHz$$
Now that's an impressive number, but that configuration would in fact result in very severe distortion of the signal. We can see this by thinking a little bit more about the path followed by the stylus: it is always touching at a point normal to the surface, so if the groove has a radius of curvature $R$ and the stylus has a radius of curvature $r$, then the path followed by the stylus will have a radius of curvature $R-r$ when it's on the inside, and $R+r$ when it's on the outside. This means that you will in fact get significant distortion when $r$ becomes a significant fraction of $R$ - we don't have to wait until they are equal. Here is another picture to show this...
The red circles correspond to the position of a stylus that "just fits" inside the sinusoidal groove; the smaller green circles correspond to the center of these circles, and show quite clearly that the radius of curvature of the path of the stylus varies drastically between the peak and the trough of the path it is following.
To determine the frequency range of this technology, we have to ask then what kind of distortion we are willing to tolerate. It becomes quite clear now that
- the frequency response varies drastically across the record, with much better response on the outer diameter (the start of the recording) than on the inner diameter (end of the recording)
- distortion is a function of amplitude (quieter is better - same product $f\cdot A$ gives same distortion)
- distortion depends on the radius of curvature of the stylus (smaller is better)
- distortion becomes a problem long before you run into the physical limit of reproduction
The fact that RIAA equalization increases the amplitude of the high frequency signal doesn't help much - but then it is not defined beyond 20 kHz, so you could decide to flatten the response further out.
A couple of other things worth noting: if you have a stylus with a finer tip, you will get lower distortion at the expense of much higher pressure between the stylus and the record; this means you need to use a lower stylus pressure in order to prevent excessive damage to the record. Lower stylus pressure means that your stylus is more susceptible to "jumping", and this in turn means that the record player needs to have better vibration isolation. All these things that audiophiles do actually have some physical basis...
There is not really a "short" answer here. Obviously, as was pointed out by others, there have been attempts to make records that can reproduce frequencies much beyond 20 kHz. I hope that the above shows some of the physics in play; there is no reason why one cannot record (and reproduce) these frequencies, but it wouldn't be very easy.
It occurs to me that if you know the shape of your stylus, you could calculate the shape of the groove that you need in order to get distortion-free response. You would "pre-distort" the signal, in much the same way that the RIAA equalization "pre-boosts" the high frequencies. As long as you know the characteristics of the reproducing system, you can compensate for some of it.