Am I correct to say that when a human lifts a dumbbell from, say, 2 feet off the ground to 6 feet off the ground, he will have increased the potential energy of the weight and thus will have burned calories to supply that energy increase?
Yep, you are correct in that. When you lift a weight, the energy has to come from somewhere, and in fact it comes from chemical energy stored in your body, released through the process of metabolism a.k.a. "burning calories."
Will two different people, Arnold Schwarzenegger in his prime and a scrawny kid for example, expend the same amount of calories moving the dumbbell from 2 feet straight up to 6 feet? Even though it's much easier for Arnold to lift it up, is he still burning the same number of calories as the scrawny kid?
That's where it gets complicated. Arnold and the kid will both put the same amount of energy into the weight, but that doesn't mean they're going to expend the same amount of energy, because not all the energy they expend goes into raising the weight. Some of it goes into producing heat, some probably goes into increasing blood flow, some goes into grunting... the point is, lifting a weight goes along with a lot of other processes that each require some amount of energy.
This energy loss can be quantified by talking about the efficiency:
$$\text{efficiency} = \epsilon = \frac{\text{useful work done}}{\text{total energy exerted}}$$
In this case, the "useful work done" is the amount of energy put into the weight to move it from the ground up to height $h$, namely $\Delta U_g = mgh$. The "total energy exerted" would be the calories you had to burn to do it, which will be more than $\Delta U_g$ because your muscles are less than 100% efficient. When you pull or push on something, your muscle cells actually go through very fast expansion and contraction cycles, instead of contracting smoothly, and in that process they consume a lot of energy that doesn't go into pushing/pulling the object. (This is also why you get tired holding an object still in midair, even though it technically doesn't require any energy to do so)
The bad news is that $\epsilon$ is impossible to calculate for a system as complex as the human body; you have to measure it. In the example you're asking about, it depends on many factors, including your muscle development and how quickly you lift the weight. That's part of why Arnold would have an easier time lifting the weight than the kid: his muscles are more efficient, so he doesn't have to expend as much energy to get a given result. The same goes for lifting the weight quickly vs. slowly: the more time you take, the more contraction cycles your muscle cells go through, and the more energy they waste. (At least, that's my understanding, but I could be wrong; for better information you should check with someone who studies human physiology.)
In case you're wondering, the way this fits into the energy conservation equation has to do with the dissipation term. You may have seen the equation $E_f = E_i$ (final energy = initial energy), but more accurately it looks like this,
$$E_D = E_f - E_i$$
where $E_D$ is the amount of energy "wasted" through dissipative forces, like friction and air resistance and whatever goes on at the micro level in muscle cell contraction cycles.
For a person lifting a weight, the relevant forms of energy are chemical energy stored in the body, $U_c$, and gravitational energy stored in the weight, $U_g$. So you could write this:
$$E_D = U_{cf} + U_{gf} - U_{ci} - U_{gi} = \Delta U_c + mgh$$
The energy expended is $-\Delta U_c$, and the useful work done is $mgh$.
