Yes, the derivative of a step function is a Dirac delta. You can see this by integrating the delta function:
$$ \Theta(x)=\int_{-\infty}^x \delta(x') \mathrm{d}x'$$
where
$$\Theta\left(x\right)=\begin{cases}
1 & x>0\\
0 & x<0
\end{cases}$$
(note that $\Theta(0)$ is not defined by this prescription. If you use a symmetric representation of the $\delta$ function you'll get $\Theta(0)=\frac{1}{2}$, but that's not important right now.)
The more physical way of thinking about it is to smooth the potential out to some function which goes from zero to the maximum in some finite distance. For example:
$$V(x) = \frac{V_0}{2} \left[1 + \tanh\left(\frac{x}{\ell}\right)\right]$$
which looks like this:

In this case the width of the potential step is of the order $\ell$. The force is the derivative of this:
$$ F = - \frac{V_0}{2\ell} \mathrm{sech}^2 \left( \frac{x}{\ell} \right) $$
which is everywhere finite, but becomes very large near $x=0$ in the limit $\ell\rightarrow0$.
This makes the classical mechanics problem nice and well defined. You can integrate the equations of motion for a classical particle in this potential and see what it does. You will find that if its kinetic energy is less than $V_0$ it will have a turning point and reflect back to $x=-\infty$. On the other hand, if its energy is greater than the barrier it will continue on with a final velocity determined by energy conservation:
$$ \frac{1}{2} m v_f^2 = \frac{1}{2} m v_i^2 - V_0 $$
This holds even in the limit $\ell\rightarrow 0$. The interesting differences in the quantum theory are barrier penetration and finite probability of reflection even above the barrier.
Just for fun I've plotted some streamlines, picking some simple numbers for $m,V_0,\ell$. If you haven't seen enough classical mechanics yet this is called a phase plane. The two axes are the two variables $x$ and $v=\dot{x}$, and the curves show how they change with time. You can see that particles which come in from the left reflect if they don't have enough energy, but pass through to the right with a velocity decrease if they go ever the barrier. All particles incident from the right pass through the step (and in fact gain velocity in the negative direction). Plots like this are a great way of gaining insight into classical mechanical systems.
