Newton's bucket is a thought experiment, not a real experiment (although it is also easy to do). The point of the experiment is that the notion of "rotating" or "not rotating" is not measured by the relative distances between objects, but by the relative motion of objects relative to a space-time structure.

When a bucket is rotating, the distance between any two atoms is the same in Newtonian mechanics, so that the geometrical relationship between any two particles is no different than when it is at rest. But when it is rotating, the water is pushed out by a centrifugal force, and when it is at rest, there is no centrifugal force.

Newton concludes that the centrifugal force is a property of the motion of the bucket relative to a notion of absolute non-accelerated space. This conclusion is already present in the structure of Newton's mechanics, where the accelerations are physical forces, and they are only physical forces relative to an inertial frame--- a description of space where the coordinate axes are given at any time, so that the axes can rotate and move at a constant velocity relative to another inertial frame, but they can't accelerate. This description is not completely circular, because it is asserting the existence of rotation and Galilean symmetry, and Newton's laws of motion have this symmetry.

In Special Relativity

Newton's bucket is clarified somewhat in special relativity. When viewed in space-time, with time considered vertical, the motion of points on a stationary bucket make vertical lines that trace out a cylinder in time. When you rotate the bucket, the trajectories of the atoms of the bucket make spirals in time. Even in Euclid's geomety, spirals are different from straight lines, in that straight lines have no curvature and are minimum distance paths between two points.

In special relativity, the reason that a rotating bucket is different from a non-rotating one is obviously pure geometry. The time-paths of the atoms in the rotating case are curved, while the time-paths of the non-rotating bucket atoms are straight. The relative distances between points are not just the relative distances in space, but in space-time, and these are not invariant under rotation.

So extending the notion of distance to space-time makes the choice of reference frame no different from the choice of coordinate axes in geometry. This removes a lot of the philosophical bite of Newton's bucket, because the only reason you expected things to be the same is because of the three dimensional distances were preserved. When you see that the four-dimensional distances are not preserved, why would you expect that everything must stay the same?

Mach's Principle and General Relativity

But for Einstein, this was still not enough. The issue left is that the notion of geometry, the notion of perpendicularity and distance, is something that is acting on material objects, but is not acted upon. This is inconsistent with Newton's third-law philosophy, anything that acts must in turn be modified by its action.

That this is unnatural was pointed out by Ernst Mach. Put Newton's bucket outside in a field at night. When the bucket is rotating, the distant stars are whirling around the bucket in its frame, while when it is still, the distant stars are still. Why should the two notions of rest coincide?

In other words, Mach is asking why the local notion of "at rest, no rotation" should coincide with the cosmological notion of "at rest, no rotation". By asking this question, he is implicitly allowing the local notion of "no rotation" to vary from point to point, and then he is asking "what dynamical law makes the no-rotation frame at any one point coincide with the global notion determined by far away matter?"

Mach concludes that the matter determined the local frame. He concluded that if Newton made the bucket "Many leagues thick", that there would be some effect between the water and the bucket which would make the water follow the bucket motion, so that the centrifugal force would be relative to the bucket, and not relative to the distant stars.

This idea was uppermost on Einstein's mind when formulating GR. Einstein realized that the observed force of gravity allows for an ambiguity in the notion of rest, since it is impossible to distinguish between acceleration and a local gravitational field. So he introduced a dynamical geometry, whose dynamics would produce gravity, and so on to GR, this story is well known.

Frame Dragging

When GR was finished, Einstein asked again about the bucket. If you place a bucket in space-time, there would be a gravitational analog of a magnetic force from the walls of the rotating bucket on the water inside. When the bucket is rotating relative to the local notion of no-rotation, this force will pull the water outward a tiny amount. The force is consistent with Mach's interpretation, when the bucket is rotating and the water is not rotating along, the gravitomagnetic force pulls the water out into an immeasurably tiny parabola.

If the bucked is made more massive, the effect is enhanced. In the limit that the bucket is about to collapse to a black hole, the exterior decouples, so that the water follows the bucket only. In this way, General Relativity incorporates Mach's principle. For a bucket sufficiently massive, the water's shape will be determined by its motion relative to the bucket, not relative to the distant stars.

This effect is called "Frame dragging", and it is a well known prediction of GR which has either already been observered or is soon to be observed.

Cosmological Mach's Principle

But there is a second notion of Mach's principle, that the motion of the bucket is relative to distant matter. This notion seemed for a long time to be incompatible with GR.

There are bogus arguments in the literature regarding this. One argument is that the Kerr solution is a counter-example to Mach's principle, because it describes a black hole rotating in empty space-time, and what's the black hole rotating relative to?

Of course this is nonsense, since you can scale solutions to become arbitrarily small in classical GR. So if you have any rotating object, rotating relative to distant matter, you can scale the solution to become infinitesimal, and send the objects it is rotating relative to to infinity. Then the object will rotate relative to boundary conditions at infinity.

The real question is for situations where there are no boundary conditions. For Einstein, this meant a closed compact universe, like a sphere. In this case, he spent much time trying to establish that Mach's principle would hold precisely, so that the rotation of objects would be relative to distant objects.

The problem is that he needed a stable spherical universe first! A static spherical universe didn't work as a solution of GR. So Einstein considered a universe with a uniform dust of matter all over, and added a cosmological constant to stabilize the universe, and studied this solution as a model cosmology.

But this cosmology is unstable too. The matter will form black holes, the black holes swallow each other, and eventually, one of the black holes becomes cosmological in size. The biggest black hole turns inside out to become a cosmological horizon, and eventually swallows all the other black holes, leaving only a uniform deSitter space.

The deSitter space is the universe of inflation--- it is constantly spreading out, in a process that can be thought of as matter constantly falling into the cosmological horizon that surrounds any observer. The cosmological horizon stays a certain distance away from any observer, and this is the only stable universe of positive cosmological constant.

In deSitter space, the cosmological horizon is a form of matter--- it is continuously linked to the largest black hole in the Einstein static universe. Further, once deSitter space is empty, this horizon cannot rotate, it can only rotate if there is something in the interior (deSitter space is unique). This means that deSitter spaces, like Einstein static universes, also obey Mach's principle, so long as the notion of matter is expanded to include cosmological horizons.

Modern Answer

The modern answer is that Newton's bucket is rotating mostly relative to the cosmological horizon, and partly relative to the distant stars. If all the stars and the cosmological horizon were rotating, we would rotate with them, so that there would be no centrifugal force (this is difficult to even say in GR, because there is no coordinate independent way to set the distant stars and cosmological horizon in a rotating motion, although one can do it for the stars alone, using Godel's universe, so long as the horizon becomes inconsistent).

This is the modern view of Mach's principle. This point of view is obvious given modern holographic physics, but is not in the literature. It is a complete resolution to the problems raised by Mach's principle, so that there is no mystery left. Mach's principle in this point of view, is just a stunted classical precursor to the holographic principle. It is asserting that all motion is to be measured relative to distant horizons.

The cosmological horizon is the distant matter in a deSitter space--- all rotation can be considered relative to the cosmological horizon.