When you touch something initially at room temperature there will be heat transfer away from your skin to the object because the temperature of your skin is higher than room temperature. How "cold" each object will feel depends primarily on the *heat transfer rate* from the skin as that determines how quickly the skin cools. That, in turn, will depend on the thermal conductivity of the material being touched. Rock and metal have higher thermal conductivity than plastic or wood and thus will feel "cooler".

Another factor to consider is the heat capacity of the object. Aluminum foil has high thermal conductivity but low heat capacity for the area touched. So aluminum foil at room temperature will not feel as "cool" as a chunk of aluminum at room temperature, even though they have the same thermal conductivity. 

Hope this helps.