In a room at normal room temperature, certain materials, such as metal, glass, ceramic, or rock, will feel cold to the touch, but others, such as wood or plastic, do not so much.
When you touch something initially at room temperature there will be heat transfer away from your skin to the object because the temperature of your skin is higher than room temperature. How "cold" each object will feel depends primarily on the heat transfer rate from the skin as that determines how quickly the skin cools (decreases in temperature). That, in turn, will depend on the thermal conductivity of the material being touched. The thermal conductivity of rock and rock like materials (k = 0.92 W/m.K$m\cdot K$), fall between the extremes of metal (203 W/m.K for aluminum) and plastic (average of about 0.25 W/m.K). It will also depend on what area of skin is in contact with the object. The object will feel cooler on the skin of the forearm (very thin skin) than the finger pad (thicker skin).
Another important factor to consider is the heat capacity of the object. If the objects of different thermal conductivity are all massive relative to the skin, then thermal conductivity alone determines the coolness. However, if the heat capacity on a given object is low compared to the skin that may determine the relative coolness.
For example, aluminum foil has high thermal conductivity but low heat capacity for the area touched. So aluminum foil at room temperature will not feel as "cool" as a chunk of aluminum at room temperature. Even though they have the same thermal conductivity, the amount of thermal energy available from the foil, because it is so thin, is much less and its temperature will rise quickly to that of the skin. For that matter, the aluminum foil may not even feel as cool as a chunk of rock, even though its thermal conductivity is higher, because its heat capacity is so low.
Hope this helps.