Projectile/orbital motion over very long distance We know the optimum angle for greatest horizontal displacement when launching an object with projectile motion is 45 degrees. 
How to solve the angle when it is real long distance around the earth where we cannot assume that gravity is constant but is given by $F=\frac {-GMm}{r^2}$. I use differetial equation to solve it and obtain 45 degrees. 
First, for the vertical distance $y$, change $v=u-gt$ to $y'=u-y''t$. Then solve it to obtain a function $y(t)$. When $y'(t)=0$ we have the time for maximum vertical distance. Then insert the value to horizontal equation. 
Can anyone guide me through this problem because I am not confident with my working
 A: This is far from an easy problem. In constant $g$ we determine the total flight time from $v_{0,y}$ (which contains the launch angle) and $g$, then insert it into $v_x(t)$.
So let's see if we can do that here.
$$F=\frac{GMm}{y^2},$$
where $y$ is the distance to the centre of the Earth. The equation of motion becomes:
$$ma_y=-\frac{GMm}{y^2}$$
$$\frac{dv_y}{dt}=-\frac{GM}{y^2}$$
Apply chain rule to LHS:
$$\frac{dv_y}{dy}\frac{dy}{dt}=-\frac{GM}{y^2}$$
$$v_ydv_y=-\frac{GM}{y^2}dy$$
Integrating gives us the maximum elevation $y_{max}$ achievable:
$$\int_{v_{0,y}}^0v_ydv_y=-\int_{R}^{y_{max}}\frac{GM}{y^2}dy$$
$$-\frac12 v_{0,y}^2=-GM\Big[-\frac1y\Big]_{R}^{y_{max}}$$
$$-\frac12 v_{0,y}^2=GM\Big[\frac{1}{y_{max}}-\frac1R\Big]$$
$$y_{max}=\frac{2GMR}{2GM-Rv_{0,y}^2}$$
Where $R$ is the radius of the Earth. Or for an arbitrary height $y$:
$$\frac12 v_{y}^2-\frac12 v_{0,y}^2=GM\Big[\frac{1}{y}-\frac1R\Big]$$
$$\frac12 v_{y}^2=\frac12 v_{0,y}^2+GM\Big[\frac{1}{y}-\frac1R\Big]$$
$$v_y=\sqrt{v_{0,y}^2+2GM\Big[\frac{1}{y}-\frac1R\Big]}$$
$$\frac{dy}{dt}=\sqrt{v_{0,y}^2+2GM\Big[\frac{1}{y}-\frac1R\Big]}$$
$$\sqrt{\frac{(v_{0,y}^2R-2GM)y-2GM}{Ry}}dy=dt$$
Although this can be integrated and has an analytical solution, its solution cannot be made explicit in $t$. So a vertical flight time cannot be easily obtained here.
A: Provided that the launch velocity $v$ is less than the escape velocity, the trajectory will be part of an ellipse (in red in diagram below) with the centre of the Earth at one focus (F). The shape (eccentricity $e$) and orientation $\theta$ of the ellipse are determined by the launch velocity $v$ and angle $\alpha$ (here measured relative to the vertical). The launch and landing points (A, B) are intersections of the ellipse with the circle of the Earth. The range (distance along surface of Earth between A and B) is $2R\theta$. 

The equation of the ellipse is
$r=\frac{a(1-e^2)}{1-e\cos\theta}$.  
The semi-major axis $a$ of the ellipse can be found from the Vis Viva Equation
$v^2=GM(\frac{2}{R}-\frac{1}{a})$
$\frac{1}{\rho}=2-\frac{Rv^2}{GM}$
where $R$ is the radius of the Earth and $\rho=\frac{a}{R}$. 
The radial and tangential velocity components are $v_r=v\cos\alpha, v_{\theta}=v\sin\alpha$. Substitute $a,v_r,v_{\theta}$ into equations #12, #17 and #20 in the link below to obtain 2 simultaneous equations relating the eccentricity $e$ of the orbit and the position angle $\theta$ of the launch point :
$v_r^2=\frac{GM}{a}\frac{e^2\sin^2\theta}{1-e^2}$
$v_{\theta}^2=\frac{GM}{a}\frac{(1-e\cos\theta)^2}{1-e^2}$.
which become
$(2\rho-1)\cos^2\alpha=\frac{e^2}{1-e^2}\sin^2\theta$
$(2\rho-1)\sin^2\alpha=\frac{1}{1-e^2}(1-e\cos\theta)^2$.
Solve these 2 equations (perhaps by trial and improvement, or some other numerical method) to find angle $\theta$.
Reference : Physics Pages : Velocity in an Elliptical Orbit
A: If the velocity is high enough for the projectile to escape earth's gravity then the distance can be assumed to be infinite.
Let's consider the case where the velocity of the projectile is not high enough to escape earth's gravity. Then the trajectory of the projectile is an ellipse with the earth at one of the two foci.
If the perigee distance of the projectile is greater than or equal to the radius of the earth, then the projectile will orbit earth forever.
If the perigee distance of the projectile is less than the radius of the earth, its trajectory intersects the perimeter (surface) of the earth at two points. The angle you are looking for is the angle of intersection between the two curves.
A: A bit late, but I'll share my (geometrical) answer to this.
General case
A Kepler orbit is described by an ellipse with the COM of the planet at one of the foci. I made a drawing of the solution here:

The nice thing about this scenario is that we can just minimize the orbital energy (as we are starting with no velocity, this will be the optimal orbit in terms of required launch speed $v_0$). The orbital energy per unit mass is $$\epsilon=-\frac{GM}{2a} \overset{t=0}{=} -\frac{GM}{R} + \frac{1}{2}{v_0^2}.$$
Hence, the best orbit is that of an ellipse that minimizes semi-major axis length $a$. It should go through launch point $L$ to destination $D$, which has one focus on the COM of the planet $C$.
Denote the foci of the ellipse $C$ (center of the planet) and $F$. For the ellipse, we know that $CL+LF = 2a$ ($CL$ is the distance between $C$ and $L$, and likewise for $LF$ etc.), and $CD+DF=2a$. So the second focus point must satisfy $$LF-DF=CD-CL$$
A hyperbola, with foci $L$ and $D$, also shown in the figure. The ellipse's semi-major axis length is $a=\frac{1}{2}(CD+DF)$, so to minimize this we must minimize distance $DF$ on this hyperbola (CD is fixed by the problem), so $F$ must clearly lie on the line $LD$, as shown in the figure. 
An ellipse has the property that any rays (light rays for example) that come from one focus reflect to the other focus. In other words, the angle bisector is perpendicular to the ellipse. Using this, we get an expression of the angle: $\frac{1}{2}\gamma+\beta+\alpha=90^\circ$. Next to this, we have that $\gamma+\beta=90^\circ$. Combining these, we find the optimal launch angle: $$\alpha = \frac{1}{2}\gamma$$
The velocity you need is also easily calculated, since the semi-major axis length is just $4a = (CL+LF)+(CD+DF) = CL+LD+CD$, and hence the velocity is $$v_0 = \sqrt{\frac{GM}{R}}\sqrt{2-\frac{4R}{CL+LD+CD}}$$
Destination and launch both on the ground
In the case that $D$ is also on the ground ($CL=CD=R$), the formulae become nice enough to work out further. Suppose the great-circle distance between launch $L$ and destination $D$ is $\Theta$, then the optimal launch angle will be $$\alpha=45^\circ-\frac{\Theta}{4}$$
The velocity will be
$$v_0 = \sqrt{\frac{GM}{R}}\sqrt{2-\frac{4}{2+2\sin{\Theta/2}}} = \sqrt{\frac{GM}{R}}\sqrt{2}\sqrt{\frac{\sin{\Theta/2}}{1+\sin{\Theta/2}}}$$
A nice result: as $\Theta\to 0$, the optimal launch angle will be $\alpha = 45^\circ$, and $v_0^2 = \frac{GM}{R} \Theta$, and with distance $D =\Theta R$, this reduces to the familiar $v_0^2 = g D$ (for constant gravity). 
For $\Theta\to180^\circ$ (the other end of the planet), the optimal angle will be $\alpha\to0^\circ$, with velocity $v_0=\sqrt{\frac{GM}{R}}$ (just enough to stay exactly at ground level)
