The key for producing a nice tone, is a mix of two facts: first, offering the air stream a somehow geometrical regular hole, where a stationary wave can be born with a certain frequency and other possible frequencies are filtered out, and second, a low enough stream velocity, so that no turbulences can happen and the flow is ordered (laminar).
The whistle is produced as an intermediate solution between having a hole that is small enough for the precise stationary wavelengths to happen, and big enough for the air not to flow at much too high a speed. That is why people with their fingers in their mouth can whistle louder, because, by putting the fingers, they create a richer and bigger aperture than a simple, small hole and so air can flow slowly enough but, at the same time, the fingers combined with the lips maintain the involved geometrical distances conveniently small, so that the stationary waves can happen.
When you blow too strongly or through an irregular-shaped hole, air flows in a disordered way, thus vibrating with thousands different modes, and that is why you hear the typical "hiss" sound, because a hiss is a mix of lots of frequencies (technically called White Noise).
Physics is not physics without, at least, a little maths. So now, follow me. You surely know Newton's second law:
$ma = F$
First, we assume that we take the force per unit volume, so that we use the density instead of the mass. And, at the same time, we write the acceleration as the derivative of the velocity:
$\rho \frac{dv}{dt} = f$
Because, as you know, the acceleration is the amount of change of the velocity per unit time. When we deal with fluid mechanics, that amount of change is due to two terms: one deals with the change in time of the velocity in a fixed point of space, and the other is due to the change of the velocity from one point to another. That is written in this way:
$\rho (\frac{\partial \mathbf{v}}{\partial t} + \mathbf{v} \cdot \nabla \mathbf{v}) = f$
(For those of you who see it for the first time, don't worry about the triangle, it is kind of a sophisticated derivative. Just follow what the terms mean)
The $f$ as you know, stands for forces (due to weight, springs, etc). In fluid dynamics, we like to give a special role to two kind of forces, so that we make them appear separately in the equation:
$\rho (\frac{\partial \mathbf{v}}{\partial t} + \mathbf{v} \cdot \nabla \mathbf{v}) = -\nabla p + \frac{1}{\mathrm{Re}} \nabla^2 \mathbf{v} +f$
The term with the $p$ is due to the change in pressure from one point to another. The other one with the $\mathrm{Re}$ is due to the viscosity, i.e. a measure of strong the fluid tries to avoid changes in shape (honey has a higher viscosity than water). This is the Navier-Stokes equation for an incompressible flow (the air is compressible, but in the range of speeds involved in a whistle, it can be very good approximated by this equation - for the purists: the Navier Stokes equation is experimentally found to hold, up to a good degree of approximation, in turbulent flows too).
The $Re$ stands for "Reynolds Number". It is a somewhat heuristic quantity that goes proportional to the velocity but inversely proportional to viscosity. It depends too on the geometrical dimensions of the problem, so it is not straightforward to derive its value. The important fact when you whistle is that, if you blow strongly, and thus increase the velocity of air, therefore having a big Reynolds Number, then the forces due to viscosity become less important in the equation (because the viscose term has $\mathrm{Re}$ in the denominator). The viscose forces are the ones that most help maintaining the flow geometrically ordered (laminar). When their contribution is not dominant, the flow becomes disordered (turbulent). A turbulent flow has no mechanical properties that are stable enough in time for a stationary wave to establish, so that your nice whistle vanishes and is replaced by a randomly fluctuating mix of thousands frequencies, that is, the white noise of the hiss...
