How do aerospace engineers choose a landing system? (Curiosity rover) The Sojourner rover with the Mars Pathfinder used a entry, descent, and landing system involving airbags to land on Mars.
The Spirit and Opportunity rovers each used more-or-less the same system involving airbags to land on Mars.
The Curiosity rover with the Mars Science Laboratory (MSL) used a very different landing system, including "seven minutes of terror".
Why did NASA engineers select such an apparently complicated landing system for MSL?
How do rocket scientists calculate the impact forces for a Mars landing for a given proposed landing system, when no one has ever used that landing system on Mars ever before?
How do aerospace engineers choose a landing system?
 A: Have my degree in space engineering (that's space, not aerospace; no airplane stuff in my learning) so I figured I should give this answer a go.
The NASA engineers plan this type of system thusly: After deciding on the high-level mission parameters, they make a list of several different landing systems.  For MSL, this included the final design, aerobraking, airbags, and a host of other possibilities. As time goes on, they strip away the less feasible ones (usually there is a slot named "new tech" that allows for new technology that we haven't thought of; this is usually stripped off early as we require a few years to ensure new tech is TRL8). Missions like MSL are usually in the works for a decade or so before launch, which gives lots of time for refining the mission parameters.  At some point, it was determined that the mass of MSL was going to be too large for purely aerobraking deployment. Additionally, they decided that the mission-critical systems were too delicate for airbags, which can seriously affect finely calibrated equipment (the heaviness also had to do with the airbags being scrapped). Needless to say, one-by-one the alternative landing systems were eventually all discarded as unfeasible, not conforming to mission requirements, impossible, too expensive, or too risky (everything has a risk analysis associated with it. If anything puts one toe over the determined "too risky" line, it's gone).  As mentioned, many of these alternatives were eliminated because of copious amounts of simulations that showed they wouldn't work.  And I can attest that there were many simulations.  The first (and later, the last) thing that the engineers do when they are given a new set of parameters or a new mission concept is to model it to determine feasibility, budget restrictions, timelines, etc. And these simulations aren't restricted to NASA; everyone does them and reports interesting findings. As an example, I am a physicist now, not affiliated with NASA in the least, but for the upcoming OSIRIS-REx mission, I've already run hundreds of simulations concerning orbital patterns, scan angles, data upload rates, you name it. And I've reported those findings to NASA. For something as big as MSL, they had physicists and engineers from all over the globe running tests and feeding back relevant data. Then, the engineers at NASA can take that data and narrow down the list of usable systems.
As for how they calculate impact forces, etc. We have a pretty good idea of the atmospheric conditions, the gravity map, and other important features of Mars. The many hundreds of people doing hundreds of simulations that I mentioned cover everything. NASA uses a type of "Google Mars" to pick a number of potential landing sites and a number of potential systems and then releases that information. The simulations are then run for every system at every location using the specific characteristics of that location. For our final result, we knew the gravity map and the pressure density. Additionally, we can easily estimate thrust velocity, approach velocity, and a myriad of other parameters. Then whatever is left that we don't know, we approximate (not kidding, sometimes nothing describes reality better than blind guessing). Often, using simple Newtonian physics and some powerful computers, we can then simulate almost exactly what will happen. Of course, at some point we actually go out and test an analog model on Earth.
This all being said, there is some unknowns still. Very few things have been used on Mars before, so every time we send a new probe, we're testing a new system there. While MSL landed smoothly, not everything does. I direct attention to the Beagle II Mars lander (aptly nicknamed the Mars Polar Crasher), which impacted the surface without slowing down.  However, in all cases, we make sure that the chosen landing system is the most suited one for the job.
A: At first glance, this is more an engineering question than a physics question, lurscher's comment actually answered it: Engineers think about possible solutions and their implications, test some of them in simulations and finally test some in real life or with models.
Note that 'think about their implications' actually means doing tons of calculations, doing basic and detail engineering to etc.
However, there are physical constraints in place, dmckee and aman mentioned some. The issue boils down to scaling. The heavier Curiosity would need far heavier airbags, or far more more fuel (that has to be shipped to mars) to back in on a pillar of fire, or huge parachutes/brake shields for aerobraking.
NOTE: This can be greatly improved by eloborating on the scale issues, maybe I'll be back later. As of now, this is somewhat soft.
Edit: This is a great question for the upcoming Space Exploration SE!
A: My answer should possibly satisfy you, I think. Here is an artist's conception of Mars Science Laboratory Entry, Descent & Landing...
Curiosity is large in size as it has 10 science instruments to find the availability of life in Mars. Also it uses Multi-Mission Radio-isotope Thermal Generator for fuel. Hence, airbags cannot be used to handle such a large weight since more fuel has to be supplied to carry them and also cannot be expected for such a soft touchdown. Even Aerobraking does not help the landing as @dmckee has already told about it. Moreover, three satellites such as European Space Agency's Mars Express Orbiter, NASA's Mars Reconnaissance Orbiter & Viking Orbiter were used to study the environment of Mars for the safe-landing of Curiosity. After the entry of the Rover into Mars' atmosphere, the processes such as Parachute deploy, Separation of Heat-shield, Radar data collection, Separation from the Aeroshell & using the rocket-powered boosters named Sky Crane and even touchdown are all handled by Curiosity. Hence, NASA mentioned this as seven minutes of terror.
The rocket-powered boosters is a new type of landing system which provides enough upthrust to overcome the gravity of Mars and the force exerted on the rover during entry. This is a proposed landing system because three orbiters are out there to update each situation. 
But even after these precautions, the count-down of Curiosity landing was not appropriate as expected. The Rover landed some minutes later than the expected time due to severe problems in atmosphere during its entry.
And, I'm helpless to answer your 3rd question as others have already commented it. And yes, they're very true. Scientists have done millions of simulations and tests before taking such a risky landing system.
A: Well one of the factors that has a impact on the landing system is the size of the rover. The airbag system is feasible for rovers of small size. Curiosity rover is the size of a car so it was too heavy for the air bag method. 
Integrated landing systems are also a huge design constraint requiring their own sensors, actuators, and power source. Since landing system is used once and only once it can be considered an design advantage to separate it and lose that extra weight and liability. 
