A lot of things can affect the color of an object. As you've mentioned, absorption plays an important role in determining what part of the visible spectrum gets subtracted from the color your eyes perceive. Optical bandgap arising from the microstructure of materials determines what portion of the spectrum is absorbed. It is closely related to the electronic bandgap, electronic density of states, and E-k dispersion relation of crystalline materials. In addition, crystallinity and size of crystallites can also affect a material's electronic characteristics. For example, crystalline silicon has an indirect bandgap of ~1.1eV, whereas amorphous silicon has a direct bandgap of >1.5eV but with many defect states within the gap. The larger bandgap of amorphous silicon means thin film amorphous silicon can appear to be pink-ish when compared to its crystalline counterpart.
On the other hand, emission of light via black body radiation, and photoluminescence can also change the appearance of an object.
For black body radiation, think of the sun. A hotter object appears to be brighter, and whiter compared to a colder object. That's because the peak of radiation spectrum shifts to higher energies when a piece of material is heated. Then, the appearance of an object depends also on external conditions.
Photoluminescence is the ability a material has to emit photons at the same or different energies after absorbing light. Take a look at your highlighter pens, and you'll get an idea of how the ink in a highlighter pen absorbs light at higher energies, and emits light at a relatively fixed energy. For example, a yellow highlighter pen has ink that emits light that appears yellow.