With drag, why does it take longer for an object thrown upwards to fall back down than it does for it to reach its highest point? Why does it take longer for an object thrown upwards to fall back down than it does for it to reach its highest point? We can draw a free body diagram for when the object is falling down and for when it's going up, but what does that tell us? Is there a simple argument for this or does one need to solve a differential equation? I first learned of this phenomenon during my first semester of physics but apparently I never really understood why it occurs.

 A: The Answer
Drag takes away energy from the object's total $E_g$ (gravitational potential energy) + $E_k$ (kinetic energy). Therefore the object's total $E=E_g+E_k$ is strictly decreasing.
Compare two points of equal height on the object's arc. The first point occurs while the object is rising. The second point occurs while the object is falling. Both points have the same $E_g$ because they're at the same height. The rising point has greater $E_k$ because $E=E_g+E_k$ is strictly decreasing with respect to time. Therefore for any pair of equal heights the object is moving faster when it's rising than when it's falling.
It takes the object an equal distance to rise as to fall. Since the object is moving faster (for any given height) when it's rising than when it's falling, that means the object takes less time to rise than to fall.
Note: Some comments to the original question mentioned the idea of buoyancy. Buoyancy does not cause an object thrown upwards to fall back down slower than it went up. That's because buoyancy is a conservative force, not a frictional force. Buoyancy force is constant with respect to time. All it does is decrease the effective gravitational acceleration.
A closer look at why buoyancy doesn't break symmetry
Several people seem to be confused about the effect of buoyancy on the thrown object, so let's take a closer look at the hypothetical situation where buoyancy acts on an object and drag does not. For simplicity, I will assume that the the ball is not thrown so high that the density of air appreciably changes. (If the density of air varies with $h$ then the math is harder but symmetry remains unbroken.)
There are two forces acting on the object. Gravitational force acts downward with a magnitude of $mg$ where $m$ equals the object's mass and $g$ equals the gravitational acceleration of an object near the Earth's surface. Buoyant force acts upward with a magnitude of $\rho Vg$ where $\rho$ is the density of air, $V$ is the volume of the object and $g$ is once again the gravitational acceleration of an object near Earth's surface.
The gravitational potential energy of a buoyant object in a medium of constant density is $E_g=mgh-\rho Vgh=(m-\rho V)gh$. At any point in the object's trajectory it's total energy is equal to its gravitational potential energy $E_g$ plus its kinetic energy $E_k$.
$$E=E_g+E_k=(m-\rho V)gh+\frac12mv^2$$
Solve for $v$.
$$v=\pm\sqrt h\sqrt{2\left(1-\frac{\rho V}m\right)g}$$
The variables $m$, $\rho$, $V$ and $g$ are all constant. That makes the expression $\sqrt{2\left(1-\frac{\rho V}m\right)g}$ constant too. In other words, the velocity $v$ is proportional to the square root of the height $h$ (up to a sign). That means the speed of the object is exactly the same when the object is ascending as when the object is descending for every pair of equal heights $h$. Without drag, the speeds of the object ascending and descending are the same. The displacement of the object ascending and descending is the same. Thus, the time $t$ it takes the object to ascend or descend is identical. They're both $t=\frac{2h}{v_0}$ where $v_0$ is the initial velocity of the thrown object. (This value comes from the fact that $\Delta x=\frac12(v_0+v)t$ for objects under constant acceleration. We know the object is under constant acceleration because the two forces gravitational force and buoyant force are both constant.)
It is true that the object will take longer to ascend and descend if you factor in a buoyant force. The question is asking what causes the ascention times and descention times to differ from each other. Buoyant force does not break the symmetry between a thrown object's ascention and descention.
A: @Isusr says the reason is air drag and that buoyancy does not affect the time. @Alchimista says the reason may be a combination of both.
Air drag does negative friction work on the object. Therefore it causes the object to loose kinetic energy cumulatively on the way up and down. So as @Isusr has said, it will have less kinetic energy (and therefore less velocity) at the same height on the way down than it did at the same height on the way up, yet the potential energy is the same at the same height. So the time up will be less than the time down due to air drag friction.
REVISION:
So what about buoyancy?  Buoyancy force acts only upward, so I thought it should take longer to come down than go up with buoyancy than without buoyancy. But apparently I was wrong. I believe the jury has decided otherwise.
First, we have unanimous agreement that air drag causes the object to rise faster than it descends. That was concluded in the post that is cited as a duplicate. But that linked post did not address the potential influence of buoyancy.
Insofar as buoyancy is concerned, I am now convinced that it doesn’t cause the out bound trip to be in less time than the return trip. To me, at least, the compelling argument is that the buoyancy force only effectively reduces the weight of the object. I would prefer to think the buoyancy force effectively changes (reduces) the magnitude of acceleration due to gravity. I am looking a this  from the perspective that both the buoyancy force and gravitational force are essentially constant for the distance travelled. All buoyancy accomplishes is to slow both the outbound trip and the return trip, but in equal amounts.
I see the OP has accepted @Isurs answer, and in my opinion rightly so. I have added my own upvote to the others. Kudos to @Isurs. And by the way, although the answer regarding drag may be a duplicate, the analysis by @Isurs of the possible contribution of buoyancy was not. So the answer, in my opinion, was not a duplicate.
Hope this helps.
