Negative mass, or just the 'appearance' of negative mass? What's the acid test? Breaking news here claim researchers at Washington State University have created negative mass by the characteristic that if you push it, it accelerates in the opposite direction.
The so called negative mass is a collection of rubidium atoms forming an Einstein Bose Condensate.
My question is can one claim they have negative mass by this one characteristic? Doesn't the material at least need to be checked against gravitational forces to see if it 'falls' up? 
What is the 'acid test' that indeed these experimenters have negative mass? Perhaps they are just seeing some other mechanism of the system that makes things appear as though they have negative mass.
 A: You're right, there's some confusion due to the way this paper has been publicized.
Negative effective mass
The 'negative mass' referred to in the paper is effective mass. The idea is that while every fundamental constituent of a physical system has a known, nonnegative mass, the effective degrees of freedom of the system may behave as if they have a different mass. 
This isn't a new idea; it pops up in a lot of contexts:


*

*You can claim that an electron's mass is "really" zero, because if you turn off every other quantum field, electrons are massless. But what we call a physical electron is really a combination of excitations of the electron and Higgs fields, which does have a positive mass. Since we can't separate the two, the latter description is more useful.

*If you have a sealed, almost-full container of water, you can describe the system by the position of the air bubbles instead of the position of the water molecules. Within the fluid, the air bubbles act as if they have negative mass: if you push the water down, the bubbles go up.

*Electrons in a solid can act as if they have a mass different from the electron mass. The reason is that the electrons interact with all the lattice ions; when you push on the electron, you end up pushing on the ions too. This can either increase or decrease the effective mass, possibly making it negative. The example in the paper is most like this, though it's in a BEC instead.


Effective vs. fundamental mass
Is a negative effective mass "really" a negative mass? On a fundamental level, we think of mass as the thing that goes into $E = mc^2$; alternatively it's the mass of an object that determines how it couples to the gravitational field. If you're thinking of mass this way, then no, none of the examples I listed above have negative mass, nor does the paper.
But if you're in the business of atomic physics or condensed matter physics, it doesn't matter, because relativity is totally irrelevant to your experiments. The energies are low enough that the speed of light might as well be infinite, and the excitations you're studying really do have a preferred reference frame (the lab frame). If you're a fish that never leaves the water, it makes perfect sense to call an air bubble 'negative mass', even if people outside the water disagree.
Does negative mass fall down?
You also asked whether an object with negative mass falls up or down. The equivalence principle tells us that gravity is indistinguishable from uniform acceleration. That means that positive and negative masses have to behave the exact same way under gravity, so negative mass falls down.
The common confusion here probably comes from the fact that an air bubble in water (with its negative effective mass) appears to fall up. This isn't actually true. If you drop a container of water containing an air bubble, the entire thing will accelerate downward uniformly, and the bubble will be stationary in the water, as required by the equivalence principle. You can see this explicitly in this video from the ISS (timestamp 1:05).
If you hold a container of water on Earth, the air bubbles will accelerate upward, but this isn't due to gravity. Gravity is pulling both the air and water down, but your hand is pushing the water up, and the water in turn pushes the air bubbles up. 
The excitations in the BEC, which also have negative effective mass, are fully analogous. If you drop the BEC, they'll fall down. If you hold the BEC still, they might as well 'fall up', but this is just due to interactions within the material, not to gravity itself.
