From a purely theoretical standpoint, the radius of the wheel doesn't matter, but heavy wheels are slow.
In an idealized scenario, the skateboard conserves energy. This means that its total energy when it gets to the bottom of the hill is the same regardless of how it goes down.
As a wheel rolls down the hill, it picks up kinetic energy. Some of that goes into its translational motion, while some goes in its rotation. The energy in the rotation is essentially wasted from the point of view of going fast. A solid cylinder or disk, for example, will move $1/\sqrt{1.5} = 0.81$ times as fast at the bottom of a hill it has rolled down as it would go if it slid down without rolling (and without friction).
If you have heavy wheels compared to the weight of the skateboard and rider, then you suffer most of this slowdown. If you have light wheels, the energy of the wheels hardly matters and you can approach the ideal sliding speed.
Next we want to know if this matters. On Wikipedia I found that rolling resistances can be about $0.01$. When you roll down an incline of angle $\theta$, you lose about $0.01 \cot\theta$ of the energy you pick up to friction. For now set rolling resistance equal to $c$ instead of the number $0.01$. For cylindrical wheels of total mass $m$ and total wheels and rider of mass $M$, a fraction $m/3M$ is used in rotational rather than translational kinetic energy. Thus, the rotational energy stored in the wheels becomes important when, roughly speaking,
$$m/M > 3 c \cot\theta$$
The question we wanted to answer was not when rolling resistance becomes important, but what size wheels are faster. So imagine that $c$ is a function of $R$, the wheel radius, and that $m = \lambda R^2$, saying that we'll consider wheels of the same density and thickness, but different radius.
If we differentiate both sides of the previous expression with respect to $R$, we get a condition for the extra rotational energy stored in the wheel size to start being a bad trade off for any improvement in rolling resistance.
$$2\lambda R/M > - 3 c'(R) \cot\theta$$
or
$$ R > -\frac{3 M c'(R) \cot\theta}{2\lambda}$$
When this inequality is satisfied, making the wheels large will slow you down. Otherwise larger wheels are better. Note that if $c'(R)$ is zero or positive, larger wheels are always worse.
Unfortunately, I can't think of good ways to estimate $c'(R)$ without experimentation. One thing I can think of is this (it's highly speculative): A skateboard has pretty hard wheels that probably don't deform much under the weight of a rider, but skateboards are treated roughly and may get some grime in their bearings. So I would guess that friction in the bearing could be the most important factor in rolling resistance for a skateboard. I don't have much experience with them, but I think that if I take a skateboard wheel and spin it while holding it up in the air, it won't spin and spin for a minute or more like a bicycle wheel will. Larger wheels mean fewer rotations and less motion in the bearing, so this would give $c(R) = \alpha/R$ for some constant $\alpha$. This guess would give
$$ R > \frac{3 M \alpha \cot\theta}{2\lambda R^2}$$
or
$$R > \left(\frac{3 M \alpha \cot\theta}{2\lambda}\right)^{1/3}$$
When I let the density of the wheels be $1g/cm^3$, the mass of the rider by $75 kg$, the slope $10 ^\circ$, the width of the wheels $2 cm$, and the rolling resistance $0.01$ when the wheels are $5cm$ tall, I get that larger wheels slow you down when $R > 17 cm$. That would indicate that larger wheels are actually better in this case up to a pretty big size wheel (for a skateboard), but take it with a large grain of salt. There are lots of unjustified assumptions in there.