How can you tell if a given reference frame is inertial? An inertial reference frame is one in which a particle has constant velocity if and only if has a zero net force acting on it.
How can one determine if a given reference frame is inertial? For example, consider an elevator falling on earth, uniformly accelerating towards the ground. There is a man inside and can see only the interior of the elevator. If he throws a ball horizontaly w.r.t to the floor of the elevator, he will observe the ball moving in a straight line, as the ball has the same vertical acceleration w.r.t to the Earth as the elevator. 
If we now place this elevator in the far reaches of the solar system sufficiently far from any planets and we repeat the experiment, the man again observes the ball moving in a straight line.
The problem is, in the first situation, the frame of reference of the man is accelerating with respect to the earth, and there is a real force acting on the ball. In the second situation there is no force acting on ball, but the behavior is the same.
The man has no way of knowing if there is a real force acting on the ball, so he can't determine whether his frame of reference is inertial, right?
 A: What you just realized in that freefall, even in Newtonian mechanics, looks locally like an inertial frame.
This is called the equivalence principle, and led Einstein to postulate that gravity is not a "real force acting on the ball".  In fact, gravity when we are standing "still" looks like the pseudo force we would observe in the elevator if we were to "place this elevator in the far reaches of the solar system sufficiently far from any planets" but have it accelerating.
Unfortunately, unlike the usual fictitious forces we encounter when studying Newtonian mechanics (Coriolis force, the centrifugal force), we can't make gravity easier to describe just by changing to a more appropriate coordinate system.  Instead Einstein found we need to consider space-time curved.
And in GR, a free-fall frame is a locally inertial frame.
So your intuition was right.
And to answer your titled question:
Any linear transformation would preserve straight lines, but could make other physics look bizarre in those coordinates. So free floating objects moving in  a straight line is not quite enough.  Having the metric be -1,1,1,1 diagonal (or opposite sign) is usually what people mean by an inertial frame.
