How to define an inertial frame of reference mathematically? I want a definition with proper chosen coordinate axis which will help me to differentiate it with the non inertial ones. I have been trying to find out what are the invariants in those with constant velocity or rest frames.
In classical mechanics an inertial frame is by definition a reference frame where the law of inertia is valid. It's a physical definition. You can then mathematically find an infinite number of reference frames requiring that inertial reference frames are frames that move with constant velocity with respect to an inertial one. So, you discover a frame is inertial with experiments, be them real or thought experiments.
This issue is addressed in the book 'Gravitation', by Misner, Thorne, and Wheeler.
Point of principle: how can one write down the laws of gravity and properties of spacetime in Galilean coordinates first (par. 12.1), and only afterwards (here) com to grip with the nature of the coordinate system and its nonuniqueness? Answer: (a quotation from par. 3.1, slightly modified): "Here and elsewhere in science, as emphasized not least by Henri Poincaré, that view is out of date which used to say 'Define your terms before you proceed.' All the laws and theories of physics, including Newton's laws of gravity, have this deep and subtle character, that they both define the concepts they use (here Galilean coordinates) and make statements about these concepts."
The discussion in section 3.1 of the book goes as follows:
All the laws and theories of physics, including the Lorentz force law, have this deep and subtle character, that they both define the concepts they use (here B and E) and make statements about these concepts. Contrariwise, the absence of some body of theory, law, and principle deprives one of the means properly to define or even use concepts.
Any forward step in human knowledge is creative in this sense: that theory, concept, law, and method of measurement - forever inseparable - are born into the world in union.
So: according to MTW, and I think their point of view is very convincing, every theory and law serves both to make statements about concepts, and as operational definition of those concepts.
And yeah, superficially that looks similar to circular reasoning.
The difference, of course, is that once we get to applied physics we have a test. You design a machine or a process, and when the design performs the way your physical laws predict then you know you are on solid ground. Example: we launch probes to other planets in our solar system; their motion matches our physical laws.
(Incidentally, that does raise the question: how about a discipline that is not in a position to apply any of its theories in the form of a device or a process? Yeah, I think in that situation it is possible to produce circular reasoning.)
I believe there is no honest "mathematical" definition of an inertial frame. I heard two definitions in my life:
1) A frame centered at Sun with axis pointing to distant objects in the Universe.
2) A frame where Newtonian laws are valid.
I believe the second one is definition in "circle" : In which frame are Newtonian laws valid? In an inertial one. What is an inertial frame? Well, the one where Newtonian lows are valid...
The first one is not "mathematical". Yet, my personal preference is the first one, but I would modify it:
An inertial reference frame is that one of Cosmic microwave background or one moving with constant speed vector to it...
How to define an inertial frame of reference mathematically?
The definition of an inertial is physical, not mathematical. However, you can easily recognize and identify them mathematically.
An inertial frame has a metric of the form $$ds^2=-c^2 dt^2+dx^2+dy^2+dz^2$$ A metric of any other form indicates a non inertial frame. A second way to identify an inertial frame mathematically is that all of the Christoffel symbols vanish everywhere.
Inertial frame can be defined as frame where the time derivative of basis vectors relative to another inertial frame is constant.
For example, Suppose an inertial frame O, if we want to know if O' is inertial or not we can say that, O' with basis vectors $<e'_1,e'_2,e'_3>$,
If inertial frame O, measures this we can say that, that frame (O') is inertial. If the result is non-zero then O' is definitly non-inertial.
It has been argued (in the answer by @RenatoRenatoRenato) that an inertial frame of reference is the one where the Law of Inertia is valid. This is equivalent to saying, as also argued (in the answer from @Dale), that the Christoffel symbols vanish everywhere.
I had a professor that said: "how do you know a frame of reference is inertial? Easy! Just take the frame that has the simplest set of forces!"
While these are obviously not incorrect, I'm afraid they do not answer the question in a practical way. As argued (in the fine answer from @Cleonis, and also by @F.Jatpil), you have to define what is an inertial frame of reference before stating the first law, in fact before stating any of the laws of mechanics, perhaps even before stating any of physics!
You always have to actually measure the system to know which frames are inertial, there's no escaping that. Ultimately, physics, differently from mathematics, is a natural science, so it's grounded on aspects of the physical world which we use as basis for defining everything else. Even though we might someday get a full explanation for inertia, that explanation would likely hinge on some other physical concept, perhaps far more profound, but its hard to believe that it would be purely mathematical in its nature.
As a side note, I like the idea from @F.Jatpil, to observe the CMB, as an accelerated frame would see that radiation "hotter", but even then you have to measure it somehow. And note that that method would still not work in the case we live in an accelerated universe.
The bottom line is: you cannot define an inertial frame of reference from first principles, you need to equip your coordinate axis with accelerometers and gyroscopes from which you measure proper acceleration, and by which you are able to distinguish it from purely coordinate acceleration.