When we move do we "borrow" momentum from the earth? In 2001 Arthur C Clarke wrote:

Like a ball on a cosmic pool table, Discovery had bounced off the moving gravitational field of Jupiter, and had gained momentum from the impact... Yet there was no violation of the laws of mechanics; Nature always balances her books, and Jupiter had lost exactly as much momentum as Discovery had gained.

Does this mean, as I peddle off on my bicycle, I am "borrowing" momentum from the earth, and when I apply the brakes, I return it?
 A: 
Does this mean, as I peddle off on my bicycle, I am "borrowing" momentum from the earth, and when I apply the brakes, I return it?

That is correct. This follows directly from the third Newton's law of motion - all forces come in pairs (action-reaction) that are equal in magnitude and opposite in direction.
As the Earth is pushing you forward, by third Newton's law of motion you are pushing the Earth backward. Since change in (linear) momentum is defined as
$$\Delta \vec p = \vec J = \int \vec F dt$$
this means that sum of changes of your and Earth's momentum due to forces between you and Earth is zero. In other words, momentum is conserved in an isolated system if there are no external forces or the net external force on the system equals zero.

Here is an excerpt from one of my earlier posts:

You can try this principle at home with roller skates and some (modestly) heavy object that you can throw. Throw the object in front of yourself while standing at rest in roller skates and watch what happens - in addition to breaking your parents furniture, you will start moving backwards due to the conservation of momentum. Roller skates are used to minimize friction with the surface, and the object you throw is like the fuel in the rocket.
From How can satellites change direction without any medium in space?

