I'm confused, does the absorption and emission determine the color of something? Or does that only happen when something is emitting energy?
When light hits an object, the photons get absorbed, then emitted with a different wavelength right?
From a quantum mechanical perspective, all light scattering is a form of absorption and re-emission of light energy. Photons don't bounce off a surface.
When the energy (proportional to frequency, or color) of the light is far from resonance with an energy transition for the material, then the the re-emission happens (nearly) instantly and the light maintains its coherence. Since the emitted light has the same energy as the absorbed light, this color is reflected by the material.
Understanding the refractive index of a material assists to understand the colors to expect under given lighting. When light is traveling through a medium it's phase is shifted according to the material's optical properties and especially the distance which light will travel inside the material. This also applies to the angle of reflection if a surface does not allow the photon to travel through it.
Another source of relevant information is the study of Spectroscopy which 'illuminates', pun intended, correlations between specific colors with specific atoms, molecules, meta-materials, etc.. Essentially, atoms each have their own signature spectra which helps identifying them in physics, astronomy and various pursuits. Spectroscopy is also used in the development of optical lattices and unique molecular structures.
Not really as in your assumption, if that happens, you can have (for example) either fluorescence or raman. The color you see is determined by the light which is reflected from the object, i.e. by the light which is missing compared to the white light. Each material only reflects certain wavelengths, and absorbs others, influencing the color you can see.