If you believe in the Hawking radiation, which follows pretty clearly from semiclassical assumptions (and via something like 20 different derivations) in a regime where semiclassicality is beyond reasonable (as in, you have microscopic quantum fields, and a stellar-mass black hole, which should preclude regime mixing), then the black hole is literally in a thermodynamic equilibrium with Hawking radiation. If you believe this, then you inevitably have to associate an entropy with the black hole, and then the second law basically becomes "the area of a black hole will increase by at least as much as the entropy of what you threw into the black hole"
If you are going to doubt this picture, then you have to doubt one of the following, in inverse order of likelihood that they're wrong:
- Thermodynamics/Stat Mech itself
- Quantum Field Theory
- General Relativity
- The semiclassicality assumption
But they are all pretty good assumptions. Semiclassicality may fail if quantum gravity has some sort of nonlinear effect that trickles all of the way up, though -- say if Hawking radiation is forbidden by some rule of quantum gravity, in the way that the decay of the Hydrogen ground state is forbidden, for example.