Is it faster underwater to go down without air or up with air? So I had a very interesting conversation between me and my friends about this topic, but we could not find the answer.
The Situation
Lets say you are in a pool and you want to touch the bottom and go up as fast as you can. You have two options - Go down without any air (suppose you can last long enough in the water...) thus creating a faster descent but longer ascent. Or you could suck as much air as possible thus creating a faster ascent but slower descent.
Lets say you push yourself down with a pole connected to the pool and kick in the bottom with your legs. Suppose your hands are as strong as your legs and both the kick and the push provides the same force. And lets suppose the pool is an average 2.5 meter deep. You swim with your hands as fast as you can down and as fast as you can up, so lets suppose these also provides the same force.
The Question


*

*Which option is faster? With or without air?


I tend more towards "with air", because it has a minor effect on your weight but it gives you more "energy or power" to do the action.
 A: I like Jerry Schirmer's answer, but I was worried that instead of modelling you swimming as fast as you can as a constant force, I thought it would be interesting to consider swimming as fast as you can as a constant power.  This seems more logical to me as if you are really trying to go as fast as you can, you will be limited by how hard you work, how fast you wave your arms about, how much power you as a human can dish out.
Since we have that $P = \vec F \cdot \vec v$, we have:
$$ ( c v_{\pm}^2 \mp \beta ) v_{\pm} = P $$
where $v_{\pm}$ is our velocity going up and down respectively, $P$ is our power output when we swim as hard as we can, $c = \frac 12 C_D \rho_w A$ is the drag, and
$$ \beta = ( \rho_w - \rho_h ) V g $$
is your net buoyant force in the water, set to be positive when you float.
The time it takes you to make the journey is 
$$ t = \frac{D}{v_+} + \frac{D}{v_-} $$
where $D$ is the depth you swim
In order to simplify this a bit and see the generality of the answer, let's nondimensionalize everything, let's write
$$ v = v_0 \nu = \left( \frac{P}{c} \right)^{1/3} \nu $$
where $\nu$ will be our dimensionless velocity, and $v_0 = (P/c)^{1/3}$ is some characteristic velocity, here chosen to be the velocity would be if we had no net buoyancy force (i.e. $\beta = 0$).
We will nondimensionalize our time as 
$$ t = t_0 \tau = \left( \frac{ 2 D }{ v_0 } \right) \tau $$
where $t_0 = 2D/v_0$ is the transit time in the case of zero net buoyancy.
Using these to simplify our equations we obtain
$$ ( \nu_{\pm}^3 \mp \alpha ) \nu_{\pm} = 1 $$
$$ \tau = \frac 12 \left( \frac 1 v_{+} + \frac 1 v_{-} \right) $$
$$ \alpha = (\rho_w - \rho_h) V g v_0 / P $$
Notice how simple our expression has become, we have a dimensionless time given ultimately in terms of just a single dimensionless parameter ($\alpha$), which itself just measures the ratio of the power consumed by the net buoyancy contribution to our swimming power.  This enables us to plot all possible solutions for all choices of parameters:

Notice that we still have the property that our minimum transit time occurs for the case where we are neutrally buoyant, and if we either want to float or sink it will cause an increase in our transit time, but notice also that this fractional increase in our transit time is small even for large values of $\alpha$.
Finally, I tried to estimate the particular value of $\alpha$ for humans with and without full lungs of air.  I took $\rho_h = 985 \text{ kg/m}^3$, $V = 66 \text{ L}$ for anaverage human body volume, but $2 \text{ L}$ more when you're lungs are full of air, as that seems to be the average vital capacity of human lungs, and $P=100 \text{ W}$ for decent effort. These correspond to the two dashed lines.
These give
$$ \alpha_{\text{empty lungs}} = 0.16 \quad \alpha_{\text{full lungs}} = 0.17 $$
$$ \tau_{\text{empty lungs}} =  1.0028 \qquad \tau_{\text{full lungs}} = 1.0032 $$
So, while technically having your lungs full takes 1.0032 times the time it would take if you were neutrally buoyant (in this model), and having your lungs empty would only take 1.0028, both of these numbers are hardly distinguishable from 1.0, and their difference from each other is hardly trustworthy given the supposed accuracy of this model.  But I thought it was interesting that a constant power swimming model, which I think is more realistic, gives an answer that depends only on the dimensionless ratio of the net buoyancy contribution to your output power. It has at its optimum the neutral bouyancy case. And it has that even for large values of $\alpha$, the transit time is hardly effected at all.
So, my advice is to take as much air as you like.
A: Without repeating the mathematics of @alemi's answer, let's just think about one other thing:
Buoyancy is a function of depth - that is, as you descend your lungs are compressed and your buoyancy decreases, then goes negative (depending on the fat content of your body - if you have enough fat, you remain positively buoyant at any depth since you do not need the lung volume to make you so).
Now if you want to dive deep (rather than just fast), it is imperative that you take a deep breath (but do not hyperventilate - that can cause you to pass out and kill you), and hold it. Why? Because your muscles will get progressively more tired as the dive proceeds; and you want to get the most assistance at the end of the dive (the return to the surface).
Second reason: your body excretes a certain amount of carbon dioxide per unit time (more during exertion). The larger the volume of air in your lungs, the slower this $CO_2$ concentration builds up. And it's the concentration that triggers the body's reflex to gasp for air.
When it is just a question of speed, you still want to be neutrally buoyant (per @alemi's answer) - so you need to figure our for your body composition whether that requires a deep breath or a shallow one. But in my experience as a diver, the arguments above trump the question of speed - you can perform far better as a diver if you move slowly and conserve oxygen.
In this case, floating at the surface then kicking your legs up above the surface as you point your body down gives you a very nice push to the bottom. Sometimes people will grab hold of a stone to help in their descent, and leave it behind at the bottom when they are ready to ascend. 2.5 m is really not all that deep. Free divers routinely can go to 30 m (the world record is almost three times that - see http://www.herbertnitsch.com/). That takes serious practice and some specialized equipment... not something I would recommend you try without professional training.
UPDATE regarding the mathematics of buoyancy. The average male human lung total volume (max inspiration) is about 6L - note that Herbert Nitsch, the free diving record holder, has a "capacity" of 10 L and tricks to expand his lungs to 15. But let's stick with "average".
At a depth $d$, the buoyancy due to a volume V_a (lung volume at 1 atm) in water of density $\rho_w$ is
$$F_b = \frac{V_a * \rho_w}{1 + d*g*\rho_w*10^{-5}}$$
For a 6 L lung at 2.5 m depth, the volume is reduced to $\frac{6}{1+0.245} = 4.8 L, so the force is reduced by 11.6 N.
An obvious question is: how does this compare to the drag on the body - is this an important factor? I found some interesting research online at http://www.fade.up.pt/docentes/leandromachado/biomecanica/Hydrod-Drag.pdf - a biomechanics lecture given at the University of Oporto in Portugal (I believe the author was J. Paulo Vilas-Boas, Full Professor and Olympic coach). He studied the deceleration of the body of a swimmer in the "gliding" position - just after the turn, a swimmer pushes off the wall, then decelerates before resuming swimming. By analyzing the deceleration, you can estimate the apparent drag force. Reproducing an interesting diagram from this talk:

It is not clear to me whether he used the "real" inertia of the swimmer here (apparent inertia of a body in water is increased by the fact that when you accelerate, the water around you accelerates as well. For a sphere, the exact solution can be computed and tells us the apparent inertia is the mass of the sphere plus 0.5 x the mass of the displaced fluid.
Either way - the approximate drag force estimated peaks at 25 N (just after the push - when velocity is highest). I do question the time scales on the graphs - they don't seem to line up quite how I would expect, but that may be due to a shift because of a numerical differentiation.  Still - with this amount of drag force, I think the best strategy is to have neutral or negative buoyancy going down, and push off the bottom to go back up - that initial push gets you almost straight to the surface (from 2.5 m) with a push that no amount of buoyancy could help you with. After all, your feet on the bottom can probably push about 500 N (conservatively - that's not even your entire body weight) which is so much greater than the buoyancy. You will have a much harder time going down (no solid surface to push) so negative buoyancy will help you.
This argument is not valid when you attempt a much deeper dive - then the biological argument I made earlier will dominate and you need the positive buoyancy for the last part of the journey; in that case I would recommend aiming for neutral buoyancy "halfway down" - using the equation above. Note that the degree of positive buoyancy you will end up with near the surface will be much greater in that case than the negative buoyancy near the bottom - again, see the formula to see why this would be so.
Example - if you wanted to free-dive to 20 m (a more serious distance), aiming for neutral buoyancy at 10 m (2 atm) with in initial lung volume of 6 L would give you a "mid dive" volume of 3 L - so a net buoyancy of 30 N near the surface, and net negative buoyancy of -15 N at the bottom of your dive. These numbers are for illustration only...
A: OK, lets solve this in as simple an approximation as possible.  I"m going to assume that the whole trip happens at the appropriate terminal velocity, and that acceleration times are very small.  Furthermore, I"m going to model the resistant force${}^{1}$ of the water as $F = cv^{2}$, for some constant $c$.  On the way down, you have the the resistant force pushing up, gravity pushing down, and the buoyant force (B) pushing up, and the swimmer's force (S) pushing down.
So, we have $S + mg - cv^{2} - B = 0$, which gives us $v = \frac{1}{\sqrt{c}}\sqrt{S + mg - B}$, which then gives us the time to travel down to be $h/v = \frac{h}{\sqrt{c\left(S + mg -B\right)}}$.  Meanwhile, for the return trip, we have the resistant force pushing down, gravity pushing down, and the swimmer's force pushing up and the buyoyant force pushing up.  This gives us $S - mg - cv^{2} + B = 0$, so we have the time to return as $\frac{h}{\sqrt{c\left(S - mg +B\right)}}$.  Thus, the total travel time $T$ is:
$$T = \frac{h}{\sqrt{c\left(S + mg -B\right)}} + \frac{h}{\sqrt{c\left(S - mg +B\right)}}$$
To investigate the effect on the travel time by increasing the buoyancy, we compute the derivative:
$$\frac{\partial T}{\partial B} = \frac{h}{2\sqrt{c}}\left(\frac{1}{\left(S +mg - B\right))^{3/2}} - \frac{1}{\left(S-mg+B\right)^{3/2}}\right)$$
If increasing buyoancy is going to make us faster, this condition tells us that the second term has to be larger than the first term, which means that its denominator needs to be smaller, so we have:
$$\begin{align}S - mg + B &< S +mg - B\\
2B < 2mg\\
B < mg
\end{align}$$
Note that there is also an implicit assumption that $S > |mg - B|$ in all of this.  So, breathing in more air will make you ascent faster to the surface only if your buoyant force is less than your weight, which is true only if you sink in water (not true for most people).  
${}^{1}$ Note that the final result isn't particularly sensitive to the details of this.  In particular, only constants and exponents, but not the final result, change for any function of the form $F_{R} = cv^{n}$ for $n\geq 1$
