Why is it so much harder to keep the same room cool than to keep it warm? I have read this question:

Let me offer a different perspective on this. Cooling is not universally harder than heating.

Why is cooling much harder than heating?
Now I have noticed that whenever in the summertime I cool a room down (which itself is very hard, takes long hours, if not a whole day), if I turn off the AC, the room warms up almost immediately, maybe a half an hour and it is completely warm again (I tried this in the summertime, when there was a heatwave, and the same happened).
Now on the other hand, in the wintertime, if I warm up the same room (pretty easy, takes few minutes, maybe a half an hour with furnace or electric heater), and then turn off the heater, the room stays warm for very long, long hours, sometimes the whole day, without cooling really off.
This room is in California, on the second floor, and the roof gets direct sunlight.
Naively, I would think that it is all thermodynamics, particles colliding, interacting, and exchanging kinetic energy. As the room tries to cool off or heat up (after turning off the heater or the AC), it should do it through the walls and windows, the particles should collide and exchange kinetic energy, until it (the air molecules) finds thermal equilibrium with the outside temperature (the air molecules outside).
Please note I am not asking why it is so hard to cool the room down with an AC in the first place or why it is so easy to heat it up with a heater. I am asking why the cool room warms up faster (loses the cold after you turn off the AC) than the warm room loosing cool down (lose the heat after you turn off the heater).
Question:

*

*Why is it so much harder to keep the same room cool than to keep it warm?

 A: 
I am asking why the cool room warms up faster (loses the cold after you turn off the AC) then the warm room loosing cool down (lose the heat after you turn off the heater).

That's not a universal thing.  That depends entirely on the room and the conditions.
But in your description, one big difference is solar heating.  When the room cools, (assuming you don't have significant drafts), most of the heat loss will be via conduction through the walls and convection outside.  Radiative losses occur as well, but they tend to be much smaller at room temperatures.
On the heating side, the same would be true for the shaded room on a hot day.  But if the room isn't shaded, you add in radiative heating.  The sun is (in some situations) simply able to deliver a lot more power to your room than your room would dump on mildly chilly day.
Besides the solar heating, it depends significantly on outside temps.  Not sure what part of California you are, some places have common summer temps of 100F where you have nearly 30F difference over preferred temp.  You mention a heat wave where it might be another 10F above that.  It might be much rarer for that location to have a day that much below temp point (say ~44F).  So you might be comparing 30F heating days to 18F cooling days.
Besides the outside, you might get heat transfer from other rooms.  You said this room is on the second floor.  Since heat tends to rise, you might benefit from any warmth in the room below, but not much from any cooling done there.
Finally, it might be a minor point in many places, but nothing running inside your home (except the A/C) is cooling it, and everything is heating it.  The refrigerator, the oven, the water heater, the dryer, the big TV, all the occupants, etc are producing heat.   For a small space (especially if you're cooking), that can make a difference.
