Do Lorentz transforms use velocity or speed? Suppose we have the $S'$ frame moving at some velocity $v$ in the $X$ direction relative to the $S$ frame, then it follows 
$$x' =\gamma (x+vt), t'=\gamma(t+\frac{vx}{c})$$
From my understanding, to get the inverse Lorentz transforms, we say that if $S'$ is moving at $v$ relative to $S$, then $S$ is moving at $-v$ relative to $S'$, then it follows, substituting $-v$ into the above equations and switching prime to unprime(with no generality lost doing this),
$$x =\gamma (x'-vt'), t=\gamma(t'-\frac{vx'}{c})$$
This relies on V being velocity not speed due to the sign change, however, on Wikipedia and such, $v$ is referred to as speed and velocity. Is this derivation flawed(yet leads to the correct answer), or is speed being used lightly?
 A: Yes, very often in physics, we use "speed" and "velocity" interchangeably out of pure laziness, as "speed" has one syllable while "velocity" has four. Lorentz boosts are always defined with velocities, though of course that is equivalent to defining them with a speed plus a direction, which in the 1D case is just a sign.
But isn't this "wrong"? Sure, but not in a way that matters. In high school and freshman physics, your teachers might spend a lot of time distinguishing speed from velocity, but that's just because it's important to drill in there is a difference. If you're speaking with people who all know that already, there's no point in being totally careful, because what you mean will be clear from context.
It's that way with a lot of stuff taught in high school. For example, in practice nobody actually cares about the "round to even" rule that means that $1.9725$ is rounded to $1.972$. That kind of edge case might not come up practically a single time in a whole lifetime, yet somehow it gets covered for a whole week in high school classes. Similarly in freshman physics you are told to always write $\int \int \int dx \, dy \, dz$ for triple integrals, because they don't trust you to remember you're doing a triple integral if you don't put all three there, but in practice almost everyone just writes this as $\int d\mathbf{x}$. 
A: 
Do Lorentz transforms use velocity or speed?

Both. If one looks at the general Lorentz transformation (boost), it's clear that both velocity and speed are used. In particular (with $c$ = 1)
$$\gamma \equiv\frac{1}{\sqrt{1-|\mathbf{v}|^2}}$$
depends on the relative speed $|\mathbf{v}|$
In block matrix form, the general Lorentz boost is:
$$\begin{bmatrix}t' \\ \mathbf{r}'\end{bmatrix}=
\begin{bmatrix}\gamma & -\gamma\mathbf{v}^T \\ -\gamma\mathbf{v}^T & \mathbf{I}+(\gamma-1)\mathbf{v}\mathbf{v}^T/|\mathbf{v}|^2\end{bmatrix}
\begin{bmatrix}t \\ \mathbf{r}\end{bmatrix}$$
