Remember that to an outside observer, 'events' within the event horizon remain forever in the future as the external observer never sees the event horizon form.
From the outside, as matter falls towards the future forming horizon, time dilation will make it appear to slow down. At the same time, the same time dilation is responsible for extreme gravitational redshift, so any light coming from these falling objects is redshifted to invisibility. In other words, falling matter disappears from sight (becomes impossible to detect) but it is still there; it never reaches the horizon, which in turn never forms.
This is what an outside observer sees. Consequently, there is no breakdown in relativity theory, everything works just fine, and apart from Hawking radiation (which is so trivially small for an astrophysical black hole, it might as well not be there) general relativity describes the physics flawlessly.
So the only way to experience an event horizon is to fall through it. It is true that an falling observer reaches the horizon in a finite amount of time, as measured by his/her clock. (But it’s an infinite amount of time as measured by an outside observer’s clock.) Once this falling observer crosses the horizon, the singularity is no longer a location in space; rather, it is an unavoidable future moment in time. Any world-line that crosses the event horizon ends at the singularity.
So the place where physics breaks down is not the centre but rather, this future moment, accessible only to observers who fell through the horizon. Physics 'breaks down' because as this moment approaches, the intensity of the gravitational field increases without limit. Since we do not like infinities, there is also the issue that eventually, the gravitational field reaches the point when it will be comparable in strength to other forces, and when its putative quantum nature can no longer be ignored.
There are some interesting questions to ponder here, including the possibility that Hawking evaporation may mean that no horizon forms in the first place… but the nature of this breakdown of physics is similar to the breakdown of physics in the very early universe, near the initial singularity (aka. the Big Bang), and there is no horizon that would hide the Big Bang from us. So never mind black holes, in the extreme early universe, when energy levels were very high and the gravitational field was very strong, we have the same problem: the quantum nature of gravity can no longer be ignored.
As there is no proper quantum theory of gravity. There are many proposals, but none really satisfactory. In fact, some people wonder if gravity is a quantum field theory in the first place. And there are very few ways to test these ideas.
The infamous BICEP2 observation of polarisation of the cosmic microwave background, presumably representing an imprint of primordial gravitational waves, was seen as a sign that gravity is a quantum theory; unfortunately, the observation was invalidated when it became clear that they were not able to isolate signal from noise. So we remain kind of in the dark: Should we look for a quantum theory of gravity? Something more radical? Or keep gravity classical? Perhaps contend ourselves with “semi-classical gravity”, which, after all, works very well in all domains accessible to us by observation or experiment?