Do the trajectories considered in newtons law have to be smooth? To me newtons law seems a bit vague. It says that the second derivative of the position vector R of mass m, is equal to a vector valued function F divided by m. A position vector is an ordered line segment between the origin ( or the point y(t0)) and the point y(t) where y is a curve. But what about the class of curves considered in this law?
 A: All laws of physics are "a bit vague." They are approximations which are valid (to a greater or lesser extent) only within a certain range of applicability.
The smoothness of a curve is a mathematical concept. Mathematical concepts are used in the modelling of physical phenomena but they are not physical entities themselves. Outside of some range the mathematical model describes physical reality with decreasing accuracy.
In Newtonian Mechanics on any given level of approximation forces and trajectories can be regarded as changing instantaneously, as a matter of convenience. On a smaller level of approximation they can be modelled as changing smoothly. But there is no level of approximation on which an instantaneous change is inconsistent with Newtonian Mechanics.
A: They do not have to be smooth. Consider trajectory such as this: 
$$
x=t
$$
$$
y=\left|t\right|,
$$
with trajectory that changes its direction abruptly at $t=0$ by 90°. This trajectory could describe object that bounces off the wall located at $y=0$.
Since in $x$ direction velocity does not change, the component of the force is constant. In $y$ direction the velocity does change abruptly from -1 to 1 at $t=0:$
$$
v_y=2*H(t)-1,
$$
where $H(t)$ is Heaviside step function. To get acceleration, you must take derivative:
$$
a_y=2*\frac{d}{dt}H(t)=2\delta(t),
$$
where $\delta(t)$ is Dirac delta function. If we assume the object has unit mass, this implies the force
$F_y=2\delta(t).$
To generalize, you could say the wall exerts the force on the object in the form:
$$
F_y=-2mv_y\delta(y),
$$
where $v_y$ is the initial velocity at the impact and work with it in the framework of Newtons laws. Often though such weird forces  are not usually treated as usual interactions between bodies, but rather as constraints of the motion (in this example that would be $y<0$ and $\left|v_y\right|=\text{const.}$) for which you try to find more suitable methods. 
