What determines if an object will stay in a planet's orbit? Say you threw an object 10 AU from a planet at a certain speed, would this object stay orbiting around the planet or would it shoot off into space? And does it depend of the speed the object when it was thrown? What about the mass of the object?
 A: I agree with the above comment. If the object was held stationary then released, then the object would simply "fall" toward the body. It would need to be given a transverse velocity to orbit or fly off. 
Orbital speed can be calculated from 
Vtransverse = $\sqrt{\frac{G*M}{R}}$ where G is the gravitational constant, M is the mass of the planet, and R is the radius from the gravitational center of the planet. 
Any speed greater than this Vtransverse would eventually fly off into space any speed below Vtransverse would crash to the planet eventually. 
A: Let's say the total energy of the system is 0 when the bodies are infinitely apart and at rest. 
Let the total mechanical energy be $M.E$
$$M.E=K.E+P.E$$
If $M.E<0$ then there will be a closed orbit(if the bodies don't collide) .
If $M.E=0$, there could be(or not) an open parabolic path depending upon initial conditions, but not a closed orbit. If $M.E>0$ there could be(or not) an open hyperbolic path, depending upon initial conditions. 
So, yes it depends upon the speed of the object, mass of the object and the orientation in which it was thrown. 
If you want to know the exact mechanism, I suggest you go through a chapter on central forces which you can find in any undergraduate textbook. 
For more detailed knowledge of orbits, you can search the web for astronomical texts but I don't think you need that. 
PS: Since I don't have enough privileges to comment, I would like to tell you that the answer given above by @Derek Wallin isn't quite correct. You can understand it like this. The transverse velocity needs to be just enough so that the object can pass the planet without collision. What this means is that the width of the planet should not come in it's path. For distances >>R, this velocity is very close to zero. This velocity would depend upon the distance between the planet and the object, angle between direction of projection and line joining their centres,radius of the planet. The expression for transverse speed he has written above is incorrect.  If the object misses collision, it can in a sense rebound and then keep on missing the collision which is what being in an orbit is all about. If you want the exact expression for transverse speed, please let me know.
