It sounds right, as no dude has a preferred frame of reference. To be sure, let's use the infallible Lorentz transformation (though I cannot, in good conscience, invert $x'$):
Suppose you remain at:
$$ E_1 = (t_1, x_1) = (0, 0),$$
while the dude starts there. After a time $T$, he has moved to $x=D$, placing that event at:
$$ E_2 = (t_2, x_2) = (T, D) $$
while you remain at:
$$ E_3 = (T, 0)$$
for which you measure a speed:
$$ v \equiv \frac{x_2-x_1}{t_2-t_1}=\frac D T$$
Transforming to his frame using:
$$ t' = \gamma(t-\frac{vx}{c^2})$$
$$ x' = \gamma(x-vt) $$
dude sees:
$$ E_1 = (\gamma(0-0), \gamma(0-0))' = (0, 0)'$$
$$ E_2 = (\gamma[T-\frac{vD}{c^2}],\gamma[D-vT])' $$
$$ E_2 = (\gamma[T-\frac{D^2}{Tc^2}],\gamma[D-DT/T])' $$
$$ E_2 = (\gamma[T-\frac{D^2}{Tc^2}], 0)'=(\gamma[T(1-\frac{v^2}{c^2})], 0 )' = (T/\gamma, 0)' $$
meaning he didn't move and his clock ticked slower. Meanwhile, you moved to:
$$ E_3 = (\gamma(T-0), \gamma(0-vT))' = (\gamma T, -\gamma D)'$$
for a velocity of:
$$ v' \equiv \frac{x'_3-x'_0}{t'_3-t'_0}=\frac{-\gamma D}{\gamma T} = -v$$
Note that in your frame, each participant measures the other's speed at the same time, while in dude's frame, you measured his speed before he measured yours.