Operationally, the way you detect an electric field is to introduce a test charge and measure the force on it.
If there is no test charge in the cavity then you cannot measure an electric field there. It doesn't matter what the field might be.
We can make a handwaving argument that it must be zero, as follows. Imagine the conductive surface is given a strong negative charge. The electrons will space themselves out to create the smallest repulsion for each of them, because it is a conductor and they can do that. Imagine that when they are spaced that way then they exert a force on a tiny test charge inside the cavity. If they do that, then they are not ideally balanced among themselves either, and they will rearrange themselves. So they must be arranged so they would not create that force.
I asserted that they can't make the test charge move without making each other move on the surface. That's plausible but not proven, which is what makes this a handwaving argument.
Notice there could still be an electric potential. Imagine a single electron inside the cavity, while the conductive surface is covered with quadrillions of spare elections. They all push on the inside electron, but the push is the same in all directions. That is not the same thing as being in a cavity with nothing pushing at it. Most ways it's the same. It isn't getting pushed anywhere.
But just as an atom at the center of the earth has no gravity but a lot of pressure....
If the electron with no electrical force on it happened to accidentally touch the conductive wall, then it would immediately move to the outside edge of the wall and jostle a place there for itself. While if there were no charges there, it would presumably continue to move randomly.
Imagine that an insulated wire travels through a tiny hole in the wall. The wire is itself uncharged, but it has a bare end in side the cavity. The free electron will be attracted to that wire, and will leave through it. (Or another will leave in its place.) Without the charge on the conducting surface, that attraction would not be there and the charge would be more likely to be absorbed by the sall than the small wire end. The potential is like a pressure.