What would be the photon's effective mass in Newton's Law of Gravitation? If we equalize the force from the Newton's Law of Gravitation to Force on a photon in a gravitional field (I don't know if there is an equation for it). What would be the photon's effective mass? (I know photons don't have rest mass.) I mean what would be the $m$'s value from the Newton's law.
 A: As @HDE226868 explained, you can calculate the photon's energy using the Plank's constant:
$$E=h\nu$$
Using the mass-energy equivalence ($E=mc^2$), you will be able to calculate the mass of the photon:
$$m=\frac{E}{c^2}=\frac{h\nu}{c^2}$$
So, you can do this, and you can even substitute this into formula of the Newton's gravitational force ($F=\frac{G m_1 m_2}{r^2}$). But what you will get, won't be what will really happen!
For example, the photons are going always with $c$. It is an experimental result. On Newton's laws, the "force" acting on the photon should accelerate or decelerate it. It doesn't happens, instead of it, the energy of the photon changes (and maybe its direction, but it will move always with $c$).
If you want to calculate the real orbit of the photon, you won't be able to do that using Newton's laws. You will have to calculate that with Einstein's General Relativity. It won't be really simple, if you now started a physicist undergraduate studies, I think you could calculate this in around a half year, if you actively searches for a such solution.
If you use Newton, and aren't in a black hole or around a neutron star, the reality won't deviate too many from your calculated result.
A: The main point of Einstein's theory was to make gravity geometric, and particles just take the straightest path through spacetime. This means that every particle will fall at the same rate, regardless of its mass. This is what's observed in experiment, and in Newton's theory seems like an enormous coincidence: why should inertia and gravitational pull be related to the same quantity? General relativity makes it inevitable.
The point of that was to show that you can't compute an object's mass from the way it moves in a gravitational field: every mass gives the same result. A heavy object moving close to the speed of light (if you could make it go that fast) would move just like a photon.
What you can try to do instead, is look at what mass is in relativity: it's defined as the energy an object has when it is not moving (with a factor of $c^2$ to convert between units). But a photon can't be brought to rest, and all of its energy is kinetic, in a precise sense that I won't explain here. That's why we say it has zero mass. But if you only want to consider some Newtonian ideas, you'll run into trouble, because a photon can never be slowed down to make Newtonian physics valid.
