What determines invariant mass? The only answer I have been able to find is that energy determines mass but photons have energy yet are still massless. Furthermore this then leads to the question of what determines invariant energy, which I would think to be mass. So in total what are the circumstances that determine fundamental attributes of particles such as mass and energy?
 A: Answer: If a particle has energy $E$ and momentum $\vec{p}$, its invariant mass is given by $m=\frac{1}{c^2} \sqrt{E^2-p^2c^2}$, where $c$ is the speed of light. 
Explanation: 
Momentum and energy of a particle change as the particle moves faster or slower.  Equivalently, the energy and momentum that you the observer measure a particle to have change as you move faster or slower.  So neither energy nor momentum are invariant, only the combination given above remains invariant.  
In particular, as a particle moves faster, both its energy and its momentum increase.  But the values increase in such a way that $E^2-p^2c^2$ remains unchanged, and thus the invariant mass is genuinely invariant.
Note that having an invariant mass of zero is perfectly fine.  It just means that $E^2 = p^2 c^2$, or $E=|p|c$.  This is the relationship between energy and momentum for massless particles like photons. 
Finally, for reference, the relationship between mass, energy, and momentum is more commonly written as $E^2 = m^2c^4 + p^2 c^2$.  This is called the energy momentum relationship.  The case where the particle is stationary, $p=0$, gives Einstein's famous formula $E=mc^2$. 
A: Very succinctly: a system's total energy, as measured from its rest frame, is the invariant mass. The rest frame is determined by the frame wherein a system's total momentum is nought.
You are probably a little confused by the photon and other so-called massless objects. Such objects are always measured to have a speed of $c$ relative to any observer, so there is no rest frame for such systems.
This succinct definition naturally leads to the formulas for rest mass determination in Yly's answer. For massless particles, the total energy is all associated with the particle's momentum. 
I suspect you may be reading too much into the word "mass". See my answer here for some more thoughts along these lines.
