I specifically have in mind arguments made in QFT textbooks in mind. There are no rigorous foundations for QFT, at least not any that can reproduce the predictions of the Standard Model. In fact, there are non-existence theorems, such as Haag's theorem, which say that what physicists are trying to do is impossible (in the Haag's theorem case, it tells us the interaction picture is untenable for QFT's). And yet there are "arguments" made in these textbooks that people seem to think "make sense" or "are convincing". How is it possible for an argument to "make sense" if it can't be made rigorous, since rigor is effectively just a clarification of an argument in to logical terms? How can progress be made in QFT using a model which is provably unworkable (such as performing derivations in an interaction picture)? How is possible to meaningful derivations with mathematically meaningless formalism?
Excellent question! One that addresses the very nature of the scientific method as applied to physics. My views here very much agrees to those of knzhou in the comments.
What is the goal of physics? It is to construct "mathematical" models that can successfully predict physical phenomena. Here I put "mathematical" is quotation marks for reasons that will hopefully become clear below.
To construct such models in physics, a physicist would use mathematics as a tool. However, the physicist is not primarily concerned with the rigours of the mathematicians. Instead, the aim is to have a unambiguous procedure for the calculations leading to predictions. The success of this endeavour is purely determined by the agreement between such predictions and experimental observations. Once such success has been achieved, the mathematical rigour of the model is of lesser importance (yet not unimportant - see below).
Of course the true mathematician can asked "how can one use such a construct if it is not mathematically rigorous?" However, the real question should be "why does it work?"
Mathematics is very versatile and flexible. The apparent inconsistencies of the model when viewed in terms of pure mathematics may be an indication that one should look at the construct in a different mathematical way. Perhaps a different set of axioms are to be formulated to build a rigorous mathematical support for the model. These are the questions that the mathematician would ask, but as for physicists, the fact that the model makes successful predictions is enough.
QFT is a complicated thing. There are many little bits that are put in by hand simply from physical considerations, things that did not follow from some rigorous derivation. However, that does not mean one cannot find a rigorous derivation for these things. Or perhaps they may look slightly different after such a rigorous derivation yet have, to good approximation, the same physical effect.
Now, perhaps this discussion gives the impression that the endeavour to place physics models on rigorous mathematical foundations is unimportant. That is not the case. In fact, the impasse in fundamental physics may be resolved by subtle changes that needs to be made in our fundamental theories. Such changes may be identified in this mathematical process. Who knows?