If two bodies orbit each other and the mass of one of them is much larger than the mass of the other body, the equations of motion can be solved exactly. In polar coordinates the solution is given by:
This is the equation of an ellipse. As the OP correctly mentions, the actual orbit of Mercury looks more like "rose petals". This is because the actual period of the orbit is slightly less than the standard $2\pi$, so after every orbit the ellipse starts again slightly earlier, resulting in these "rose petals". This effect is known as the "perihelion precession".
What causes this effect? The main contribution is purely Newtonian, and is due to the attraction of the other planets. To first-order perturbation theory one can compute the correction to the orbit, yielding the precession of the perihelion. So it is correct to say Newtonian gravitation predicts "rose petals", at least qualitatively. In quantitative terms, the prediction for how much the perihelion should shift every year is off by a small however noticeable amount. This can only be explained by corrections due to General Relativity.
For the Newtonian contribution to Mercury's perihelion precession, a very simple method to compute it is presented in the paper Nonrelativistic contribution to Mercury’s perihelion precession by Price & Rush.
However it should be mentioned that Newton never did the calculations himself, rather these consequences of Newton's laws were explored in the 19th century.