To translate your question into a more physically addressable form: We will wave a wand or using a teleportation booth to transport moon matter to the surface of the earth, in such a way that the gravitational potential energy, orbital kinetic energy, and momentum of the transported matter disappears and the matter lands on the earth's surface moving precisely the same way as you would be moving due to the earth's rotation. What happens to the moon's orbit and the earth's rotation?
Answer: the earth would continue to rotate at an unchanged rate. The earth would become more massive, and would have a stronger gravitational field, so the moon would move in closer to the earth and would move faster in its orbit as a result. If the mass were transported all at once instead of gradually, the moon's orbit would become more elliptical, with the highest point roughly at the location where the moon was at the moment when the mass was transported. The center of mass of the earth-moon system would shift toward the earth because of the mass transfer from earth to moon.
If a non-magic method were used such as a rail gun (or a really amazing catapult), the story would be a bit different. The moon's orbital velocity is a little over 1 km/sec. The Navy has tested a rail gun with a muzzle velocity of 2.5 km/s, so it's physically possible. Launching a projectile at 1.022 km/sec in a retrograde direction from the moon would let the projectile (which would at that point have zero orbital momentum) drop straight toward the center of the earth.
A good way to think about these things is to consider the extremes. Let's say we don't worry about energy conservation but insist on conservation of momentum. Now suddenly reduce to moon to the mass of a pea by using the rail gun to move almost all the original mass to the earth. The pea-moon would have to be moving at very nearly the speed of light to have that much angular momentum (with respect to the earth). We would never see it again. The earth, on the other hand, would have increased its mass by about 1.2 percent (the mass of the moon), without any angular momentum being added – which means it would have to be rotating a bit slower. That supports your intuition.
HOWEVER we really can't ignore any part of the conservation of energy and momentum; and it's important to consider carefully the mechanisms by which the system changes. For example, a slight excess or shortage of launch speed would cause the projectile to have a net negative or positive orbital angular momentum which would be transferred to the earth on impact (by hitting the "leading" or "trailing" edge of the earth at the equator as seen from the moon). So, it could either speed or slow the earth's rotation.