Dismissing all closed timelike curves from just physical arguments is hard since a lot of them are fairly benign (such as the classic timelike cylinder where two spacelike hypersurface of a globally hyperbolic spacetime are identified), but those are usually not considered very problematic since you can always just go to the universal cover without issues. Other spacetimes with non-compact chronology violating regions can also be fairly benign and are harder to disprove, short of finding out initial conditions for the universe.
The common type of CTCs argued against are the ones which can be constructed by experiment, so-called compactly generated chronology horizons. Chronology horizons are as you may know a type of Cauchy horizon leading to a chronology violating region, and a chronology horizon is compactly generated if its null generators remain in a compact region of spacetime at some point in the past (this usually means that they all stem from some closed null geodesic, called a fountain).
It has been shown  that, in a compactly generated Cauchy horizon, there are points called base points which are past terminal accumulation points of the null generators (either points on the closed null geodesics or accumulation points if the spacetime is causal but has imprisoned curves). In such a case, the Klein-Gordon equation is singular at those points, which leads the stress-energy tensor to be divergent.
This is the current big argument for chronology protection : in such cases, the perturbation to the vacuum is such that the stress-energy tensor diverges, so that the solution is meaningless : obviously such a thing would disrupt the solution before the formation of a time machine.