Why is the definition of inertial mass circular? On Wikipedia, the definition of inertial mass is:  

Inertial mass is the mass of an object measured by its resistance to
  acceleration. And, can be evaluated using $F = ma$, Newton's second
  law.

And, in the answer of this question, the viewer has also answered in terms of Newton's second law of motion.
However, I think that both these answers are circular in nature, as Newton didn't derive mass $m$ in terms of force $F$ , he derived $F$ in terms of $m$.
Another confusion I have is related to the law of conservation of momentum. I read that it was experimentally found by Newton that momentum is a "conserved" quantity, which led him to define momentum as $p = mv$ (this is the link to one of my questions regarding momentum).
But now, when I again think of this, I wonder how did he calculated "mass". To experimentally find that momentum is conserved, he must be knowing the values of mass $m$. And even if he used a scale or a weighing machine of some sort, how was he able to calculate $m$ from $F$, even when $F$ is not defined yet?
I am asking this question because I am not able to find the explanation to this anywhere. Most people just answer this in terms of $F$, which is circular. Am I making any mistake in thinking this way, I mean is there any other theory that I don't know?
 A: If you rely on  Newton's second law, the definition of mass turns out to be circular or very intricate as also the notion of (undefined) force appears therein. A better approach consists of starting from the experimental fact that momentum is conserved. In a very theoretical picture you can deal with as it follows. You have a set of bodies and you already knows that there is a reference frame $I$ such that
all those bodies simultaneously move with constant velocity therein if they are sufficiently far to each other (and sufficiently far form the other bodies in the Universe). 
This reference frame is called inertial. Its existence is the first postulate of Newton's mechanics restated here into a more modern view. 
Remaining at rest in $I$, another physical fact is the following. It is possible to associate every body with a strictly positive real number $m$ such that, if a pair of bodies are sufficiently close to each other such that their motion shows acceleration in $I$, it turns out that
$$m_1 \mathbf{v}_1(t) + m_2 \mathbf{v}_2(t) = \mathbf{\textrm{constant}} \tag{1}\:.$$
for every $t\in \mathbb R$ and for every value of $\mathbf{v}_j(t)$ -- which are not constant -- the velocities attained in $I$ during the interaction of bodies.
It also turns out that (in classical physics)
(a) $m_i$ only depends on the $i$-th body and not on the other body, say the $j$-th one,  which interacts with the former.
(b) If a number of bodies with masses $m_1,\ldots, m_N$ form a unique larger body with mass $M$, then $M= m_1+\ldots + m_N$.
It is worth noticing that (1) can be theoretically exploited to measure the value of masses with respect to the mass of a given reference body, used as unit $m_1=1$. Measuring the velocities of this reference body and the other body at two different times, we have
$$\mathbf{v}_1(t) + m_2 \mathbf{v}_2(t) =  \mathbf{v}_1(t') + m_2 \mathbf{v}_2(t') \tag{2}$$
and thus
$$\mathbf{v}_1(t) - \mathbf{v}_1(t') = m_2 (\mathbf{v}_2(t') - \mathbf{v}_2(t)) \:.$$
as $\mathbf{v}_2(t') - \mathbf{v}_2(t) \neq \mathbf{0}$ for some choice of $t$ and $t'$ (because the bodies accelerate by hypotheses), there is at most one constant $m_2$ satisfying (2). The fact that it exists is very surprising actually!
A: 
However, I think that both these answers are circular in nature, as Newton didn't derive mass $m$ in terms of force $F$ , he derived $F$ in terms of $m$.

Newton's 2nd law does not "derive $F$ in terms of $m$"; it states if force acting on the body $\mathbf F$, mass of the body $m$ and acceleration of the body $\mathbf a$ are determined independently, they always obey the relation
$$
\mathbf F=km\mathbf a.
$$
where $k$ is a number depending on the choice of units but otherwise constant in all situations. Later, unit of force - Newton - was defined to simplify this into
$$
\mathbf F=m\mathbf a.
$$
Neither of the three quantities is defined by the 2nd law, because that would mean there is no law, only a definition.
The inertial mass $m_{inertial}$ though, is defined by the equation
$$
\mathbf F = m_{inertial} \mathbf a.
$$
$m_{inertial}$ is not necessarily constant based on this definition; it is possible that it changes values depending on $\mathbf F,\mathbf a$ or other things. However for low enough speeds, $m_{inertial}$ is proportional to $m$.

But now, when I again think of this, I wonder how did he calculated "mass". To experimentally find that momentum is conserved, he must be knowing the values of mass $m$. And even if he used a scale or a weighing machine of some sort, how was he able to calculate $m$ from $F$, even when $F$ is not defined yet?

To determine mass, one does not need to know definition or value of force. It is possible to determine mass of a body as the number that quantifies amount of matter in the body in terms of a standard amount of matter. For example, body made of 2pockets of sand has mass 2 in units of pocket of sand. Or one can measure mass based on deformation of a weighing spring.
A: My answer leans towards that of Jan Lalinsky. It is not really clear what is the historical status of force with respect to the mass $m$ of an object in the second law. Some say it is tautological and others that is contingent.
Fortunately, we don't need to answer that question here to gain insights into what Newton may have had in mind when talking about masses.
First of all, we need to first acknowledge that Newton put a great deal of effort to make his 3 fundamental axioms (now called Newtons' laws) consistent and closed.
It is also important to appreciate, as I will try to show, that Newton's synthesis was compiling insights from both dynamical observations (from Galileo and Descartes for instance) and static ones that had in fact been going on for ages just for the purpose of trade of goods, architecture etc...
If you read the second law from the english translation of his Principia it basically says that:

Law 2: The alteration of motion is ever proportional to the motive
  force impressed; and is made in the direction of the right line in
  which that force is impressed

i.e. in an inertial frame of reference (only frame in which the notion of "motive force impressed" makes sense according to Newton) we have $a \propto F$. Of course, at that stage neither $F$ nor the proportionality factor are known; but if either comes to be known then the other follows.
I think nothing in the 3 laws of Newton's really forces the proportionality factor to be exactly the mass as we know it. In fact, as Jan Lalinsky stated, we just need to name the prefactor "inertial mass" $m_I$ and the combination of Newton's 3 laws will give that the state of motion of the center of (inertial) mass of any system of points in absence of external forces is following a straight line motion at constant speed (which is equivalent to the conservation of total linear momentum...and this is true regardless of the distances between the points in the system).
Such a proposition had already been made by Descartes for instance but he had postulated that the inertial mass would correspond to the volume of the body on the basis that the laws of Nature ought to be explainable with space and time only. This turned out to be incorrect and a new fundamental concept had to enter the arena.
To see this, we can simply acknowledge that Earth is pulling on a object via a downward motive force called weight and with symbol $W$.
Assuming the terrestrial frame is inertial, we can infer that $m_I a = W$.
Now, we can apply, as Newton did, Galileo's observation that 

Provided air friction can be neglected, all bodies observed from a terrestrial frame $T$ fall with the same constant acceleration of
  magnitude $g$ towards the ground

The only possible conclusion is that $W = m_I g$, where $g$ is the same for all bodies.
There is therefore a direct relationship between the weight of an object and its inertial mass. This enables one to measure relative masses via statics experiments by invoking Newton's 2nd law and this is still how masses are measured today.
To me it seems impossible to talk about masses, in the Newtonian context, without invoking statics and gravity. One can do it as I did above by relying on the practical observation from Galileo or by postulating an additional universal law; which is what Newton did with his Universal law of gravity.
This is important because the grand Newton's synthesis makes sense, in practice, only when his three laws are combined with his universal law of gravity. In fact, he tentatively tried to show that if the gravitational mass of an object was not proportional to its inertial mass, then self consistency of his theory would be lost.
