Definitions
The rate at which the temperature of a solid object of mass $m$ (kg) and specific heat capacity $\tilde{C}_p$ (J/kg $^o$C) will change $dT/dt$ ($^o$C/s) is written as below.
$$ m \tilde{C}_p\ {dT}/{dt} = \dot{q} $$
The term $\dot{q}$ (J/s) is the rate at which heat enters (to heat up) or leaves (to cool down) the object.
To start, assume the object is initially at $T_o$ and the surroundings is held at constant $T_s$.
Types of Heat Flow
Heat can enter/leave the object by radiation to a surrounding fluid (gas or liquid). The radiation rate is proportional to the fourth power of absolute temperature $\dot{q}_r \propto T^4$. The initial overall rate of heat transfer is expressed to first order as below.
$$\dot{q}_{rT} \propto |T_o^4 - T_s^4|$$
You can use this to determine the initial temperatures of the object where the cold object receives more radiation heat flow from the surroundings than the hot object looses by radiation heat flow to the surroundings. Thus, you can determine for which cases a cold or hot object changes temperature faster than a hot object in the same air temperatures. Note that a vacuum has a temperature $T_a = 0$ K and therefore has not radiation input to the object. In this case, the hot and "cold" object will both cool down.
Heat can enter/leave the object by convection when the surroundings is a fluid (gas or liquid). The rate of heat transfer is proportional to temperature difference. Using the same process as above, you will find the net rate of heat transfer by convection is determined as below.
$$ \dot{q}_{hT} \propto |T_o - T_s| $$
We learn from this that a hot or cold object will cool or heat by convection at the same rate as long as the temperature difference between the object and the surrounding fluid is the same.
Finally, heat can enter/leave the object to the surroundings by conduction in the surroundings. This happens with a temperature gradient $dT_s/dz$ ($^o$C/m) in the surroundings. A fluid does not typically sustain a temperature gradient well; the gradient dissipates and the flow is by convection. Therefore making a statement that you want to consider conduction in the surroundings is akin to saying that you define the surroundings also as a solid. The net rate of conductive flow is written as below.
$$ \dot{q}_{kT} \propto |{dT_s}/{dz}| $$
Combined Analysis
The final equation will have a form as below.
$$ m \tilde{C}_p\ {dT_o}/{dt} = \sigma A \left( \epsilon_s T_s^4 - \epsilon_o T_o^4 \right) + h_s A \left(T_s - T_o\right) - k_s A\ {dT_s}/{dz} $$
The various terms include the Stefan-Boltzmann factor $\sigma$, object area $A$, convection coefficient $h_s$, and thermal conductivity $k_s$. This equation defines the rate of temperature change of the object at any point in time. The +/- signs are set so that, when the object is hotter than the surroundings, the object will cool down (and vice-versa). It is a rather unwieldy equation that is typically solved by considering only one of the three cases (radiation, convection, or conduction) at a time. It also presumes the object heats or cools uniformly throughout. The case when you have a thermal gradient in the solid object is yet another equation.
Observations
As a general rule, the rate at which a hot object changes temperature (cools down) cannot be stated to be faster (or slower) than the rate at which a cold object changes temperature (heats up). The rate of heat transfer depends on the type of surroundings (vacuum, fluid, or solid), the temperature of the surroundings, and the type of heat transfer (radiation, convection, or conduction).
