What makes the inertial reference frames special? What makes the inertial reference frames (IRFs) special? Specifically:

*

*Using inertial reference frames is a matter of computational convenience, but in principle we could build physics without using them

*IRFs have a special place in nature, due to the underlying properties/symmetries of the universe

*We have experimental evidence that IRFs are special (which is probably a restatement of the previous option)

In my freshman physics course (long time ago), when non-inertial forces were introduced, someone remarked that we could describe all the physical phenomena without using the IRFs. The subject was then retaken in the philosophy course, where we discussed how much of physics was real and what was human invention (heliocentric vs. geocentric system in this case, although my choice at the time was fundamental particles vs. quasiparticles in the solid state.) I never gave it much thought till recently, in connection to this question, asking for delineating fictitious and non-fictitious forces. Although many arguments have been given, most of them seem to rely implicitly on the primacy of the IRFs:

*

*Relativity – in relativity a real force is a four-vector, while a fictitious force is not. This takes for granted the relativity, which holds primacy of IRFs as one of its postulates.

*Newton's laws would not hold in IRFs. Indeed, Newton's first law postulates the existence of IRFs.

*Accelerometer will respond only to a real force – that is, an accelerometer is a sensor for IRFs, but this does not say that they are special – one could design a different sensor.

 A: I think it is simply a matter of computational convenience, which is the point you make. Any  sort of reference frame can be used- it is just that some forms make calculations more straightforward. For example, if you are trying to solve a problem which has spherical symmetry, say, it might make sense to adopt a reference frame with polar coordinates.
The reason why inertial frames are so useful is that the commonly used equations of physics take a simpler form in them than they do in accelerating frames.
A: I do not have an answer and admit that I have not yet fully read in detail all answers and comments in this question but it srikes me that Mach's principle of which Einstein was a great admirer was not yet mentioned.
In short: E. Mach was of the opinion that only when other masses in the universe (fixed stars) exist a moving accelerometer can ever measure a force.
To us today this seems like a stark statement. But please consider this:
V.I. Arnold writes on pp.3-4 of his book on classical mechanics:
"Galileo's principle of relativity:
There exist coordinate systems (called inertial) possessing the following two properties.

*

*All laws of nature time at all moments of time are the same in all inertial coordinate systems.

*All coordinate systems in uniform rectilinear motion with respect to an inertial one are themselves inertial.

In other words, if a coordinate system attached to the earth is inertial, then an experimenter on a train which is moving uniformly in a straight line with respect to the earth cannot detect the motion of the train by experiments conducted entirely inside his car.
In reality, the coordinate system associated with the earth is only approximately inertial. Coordinate systems associated with the sun, the stars, etc. are more nearly inertial."
It is Arnold's last sentence that indicates the srong connection to Mach's ideas. I also take from this that Arnold did probably not want to try to give a formal definition of an inertial system since his formulation in 1. and 2. is obviously circular.
As I said: no satisfactory answer to the question what "is" an inertial system. It boils down to taking the frame that is most convenient to do calculations
in full agreement with Marco Ocram's previous answer here.
One last remark: the above link Mach's principle brings up the paper
V.Putz, A Theory of Inertia Based on Mach’s Principle.
It can be the starting point to find some more references, esp. from H.J.Treder who wrote the book "The relativity of inertia" (1972). Unfortunately I have neither the English translation of that, nor the original German edition.
A: Inertial frames are special because they have nothing special: they are not defined with respect to the dynamics of any specific object (I take 'object' as a a synonym to 'system' here).
Non-inertial frames need a reference to a specific physical system that conceptually anchors them while, as the wikipedia article says in its first lines, "Conceptually, the physics of a system in an inertial frame have no causes external to the system".
In a relational perspective, inertial frames are anchored to objects that are not in relation with other objects, so that it is undefined whether these objects are doing something ('moving') or not.
In an accelerated frame, it is clear that an object keeping the same coordinates is accelerating. In an inertial frame, it is a matter of convention to decide if an object keeping the same coordinates is moving or not moving - within the specification of the frame itself there exists no reference to tell the difference.
A: 
Using inertial reference frames is a matter of computational convenience, but in principle we could build physics without using them

Not only could we build physics without inertial frames in principle, we already have done so. Any law of physics which can be expressed in terms of tensors is independent of reference frame. The same law, with no modifications, can be used regardless of coordinates/frames. Currently, all known fundamental laws of physics can be expressed in terms of tensors.
Note, even though we can formulate the physics entirely without reference frames we will generally use them. But we are free to choose any convenient coordinates for each given problem, which often will not be inertial coordinates. With tensors we never need to adapt the laws of physics. Using any frame, inertial or not, is thus seen as a matter of convenience, even though it is a convenience that we practically never dispense with for analyzing a concrete scenario.
Furthermore, with this framework you can learn a lot of general principles without reference frames at all. One is that there is a completely frame-independent sense of an unaccelerated object and a frame-independent sense of acceleration. This is modeled mathematically using geodesics and measured experimentally using accelerometers. In this frame-independent framework only real forces exist and fictitious forces cannot even arise. Thus the distinction between real and fictitious forces is stark and only real forces remain when we dispense with reference frames.

Accelerometer will respond only to a real force – that is, an accelerometer is a sensor for IRFs, but this does not say that they are special – one could design a different sensor.

This is actually incorrect. You cannot design a sensor that will detect fictitious forces. Only real forces can be detected experimentally.
This can be clearly seen from the fact that the outcome of any sensor in any experiment will be the same regardless of whether you analyze it from an inertial frame without fictitious forces or a non-inertial frame with fictitious forces. Since any measurement has the same result regardless of the frame and regardless of the fictitious forces, there is no sensor that can detect fictitious forces. This is further reflected in the fact that only real forces produce acceleration in that completely frame-independent sense above, which is described mathematically using deviations from geodesics and measured experimentally using accelerometers.
