When does an electron release energy as photon and when as mechanical vibration? Suppose we have a compound which has been given energy (either in the from of heat or light). Now,  its electrons would absorb this energy and kick up to higher energy levels. But, it would also re-emit this energy either in the form of heat or light. My question is: what determines whether the electron will non-radiatively (transform directly into mechanical vibration energy, thus heating the material) relax or radiatively (release light) relax??
 A: As a general rule an isolated molecule that absorbs a photon will re-emit the energy as a photon. This is simply because the energy has nowhere else to go. A non-radiative relaxation will happen only if that molecule can transfer the energy to something else.
In gases non-radiative transfer can happen if the excited molecule collides with another molecule. In that case the energy absorbed from the incident photon will usually be converted to kinetic energy of the two colliding molecules. In most cases this process is so fast that a collision will almost always result in non-radiative decay. Indeed if you want to record emission spectra from gases it has to be done at low pressures to reduce the collision rate.
In liquids and solids the molecules are in close contact and non-radiative decay is the norm. It is only in special circumstances that we see radiative decay for example as fluorescence of phosphorescence.
A: You are basically asking about non-radiative electron/atom de-exitation. It is called non-radiative, because no photons are emitted. There are basically two types:

*

*internal conversion(for example fluorescence): in this case, the molecular spin state remains the same, the energy is given off to the vibrational modes of the molecule, it is transformed into heat. It can be a case of fluorescence, because initially, the energy increases the energy of the surrounding molecules (internal conversion), which then release it in the form of photons (fluorescence).

https://en.wikipedia.org/wiki/Internal_conversion_(chemistry)



*intersystem crossing: transition between two electronic states with different states multiplicity.


When a singlet state nonradiatively passes to a triplet state, or conversely a triplet transitions to a singlet, that process is known as intersystem crossing. In essence, the spin of the excited electron is reversed.

https://en.wikipedia.org/wiki/Intersystem_crossing
Now you are asking what determines whether the energy will be released as a photon or in non-radiative transition. These are QM processes, and it is all probabilities.
Now you are asking about spontaneous emission (when a photon is emitted), and what determines whether that happens, or non-radiative transition. One of the reasons is speed and selection rules. Non-radiative transitions are not subject to selection rules, and in certain cases, they can happen quickly (relative to spontaneous emission).

Transitions which do not involve the absorption or emission of radiation are not affected by selection rules. Radiationless transition between levels, such as between the excited S = 0 and S = 1 states, may proceed quickly enough to siphon off a portion of the S = 0 population before it spontaneously returns to the ground state.

https://en.wikipedia.org/wiki/Population_inversion
In certain cases, quick
A: Other answer has the key point, generally speaking atom/molecule in a condensed phase (liquid, solid) will undergo vibrational relaxation because there is coupling to other nearby atoms/molecules, in gas phase it will radiate.
A quick explanation of the general principle: there are (approximately) 2 types of transition that could happen (radiative or non-radiative), so we are interested in these transition probabilities.  Roughly speaking those depend on how "similar" the wavefunction of the excited system is compared to the system after the transition.  The other consideration is how many transitions there are available for each type.  In the solid/liquid, you can imagine that the excitation can transfer to any of the nearby atoms/molecules vibrational modes* - in other words, there are many transitions of that type available. This is not true in the gas
'* A more technical explanation is that there is a continuum of vibrational modes in the solid, so it is highly probable that the initial excitation will overlap with some region of that continuum
