If you want to be physical, you'd have to have a physical interpretation of the derivatives.
If you've already taken two derivatives you can ask yourself whether it is possible to take the gradient of those second derivatives. If so, then the second derivatives commuted, if not then the second derivatives are weird (if something wasn't weird you could take the gradient).
Note that you have a vector field, but being a vector had nothing to do with. Scalar fields like temperature or pressure can also fail to have second derivatives commute.
Plus, how do you take partial derivatives of vectors in the first place? You take derivatives of the three scalar fields corresponding to the components (with appropriate extra factors if the frame of coordinate vectors changes).
So the second derivatives don't commute when the second derivative is weird. That's pretty vague. But here is an example. If you have an electric field then the first derivative is related to the charge density, so what if you want a charge density whose gradient is discontinuous. Then certain combinations of the second derivatives of the electric field will be discontinuous so some of them will have to be discontinuous themselves.
So if you want a charge density without a second derivative, then then the electric field might not have commuting partial derivatives.
That makes sense. The first derivatives have to exist in some sense (and keep in mind that curls and divergences can exist even if the partials commonly used to make them do not) so that the derivatives of the electric field can equal to the charge distribution. The second derivatives of the electric field might not exist if the charge distribution is discontinuous, but if the charge a gradient then the second derivatives and the gradient is continuous that is good. It is not enough. However if the gradient of the charge distribution is discontinuous then some combination of second derives of the electric field is discontinuous so one of the second derivatives of the electric field must be discontinuous. However that doesn't mean the partials fail to commute, just that they might not commute.
And there are generalizations to partial derivatives (called weak derivatives) that do always commute when they give functions, but sometimes they give distributions instead of functions. And that is just their way of stopping. After all, some times you can't just take a derivative over and over again.
And to people that want to assume that everything is smooth, sometimes that causes time travel to form in a region where time travel was avoidable by not making things smooth, so forcing things to be smooth can change things in about as big a way as is possible.
That said. If you have something without a first, second, or third derivatives ask yourself: is there something with those derivatives that experimentally looks or acts the same or very close to what I have and how would I be so sure I didn't have that instead?
So if there is something with enough derivatives that is close enough to what you have, maybe that is actually what you have. The things to watch out for are whether you are making something that is sensitive to things you can't control, lack of reproducibility isn't a friend of science after all.
Keep in mind that even a lack of a regular first partial derivative of an electric field happens at, say, a surface charge distribution. So you can easily (mathematically) make a charge distribution that is is continuous and even line up the gradients to match on a surface but set it up so the second derivatives are not continuous and where the mixed partials don't agree by just shaping the charge distribution.
But that charge distribution will be one that you can only approximate in the lab. How to avoid having there be ones that do and don't have third derivatives?
Do often you say that you never know for sure what you have, that there is always some approximations. So you say you want a thing that has some derivatives and then consider all the things where it and its first m derivatives are all sufficiently close to what you imagined, then you image using some random thing from that set.
That's similar to the specifications you'd make in your lab notes, that you machine a material to be a certain size with certain error and then maybe you also want the edge to have a certain lack of wiggle to some error and maybe you want that wiggle to not change up to some error. But at some point you stopped measuring and stopped specifying and so you don't know or care what you have. If your regularly reproduce your results then that vagueness of specification didn't matter, if you can't you might find out that you don't just want the size to be within 1mm you also need the edge to not jump around from one slope to another too much, or maybe you need the slope to not change too much, if it matters you'll specify it.
Also keep in mind the distinction between a macroscopic (averaged) electromagnetic field and a microscopic one (that shoots up around every individual atom). Also, a velocity field is an averaged field too, it isn't the velocity of every water molecule in the fluid.
So the lack of commutativity is usually assumed away. Either by switching to weak derivatives, or considering fields that had certain differentiability and then considering the stuff who it and its derivatives are all sufficiently close to the one you had in mind.
Or even just noting that your mathematical function was just a model of the actual setup so details about limits at a point might just be beyond the scope of your model.
For instance, those weak derivatives are actually only sensitive to the average derivative in some finite region, they don't care about a point.