Keeping track of units I'm not quite sure how keep track of physical units when differentiaion is involved. I find it important to keep track of units because I want to be able to check the units afterwards.
I will discuss a specific case, to illustrate my confusion with regards to the units.
Consider an RC filter low pass filter. Its transfer function is given by $$
H(\omega) = \frac{1}{1+j\frac{\omega}{\omega_{RC}}}\ ,
$$
with $\omega_{RC} = 1/RC$.
Say, we want to know its step response function $s = s(t)$; i.e. the voltage $V_{out}(t)$ that we measure over the capacitor if we hook up a Heaviside unit input voltage $V_{in}(t) = \theta(t) = \begin{cases}
    0       & \quad \text{if }  \quad t \leq 0\\
    1  & \quad \text{if }\quad t >0 \ .
  \end{cases}$
It can be shown that $s$ is given by
$$
s(t) = RC(1-e^{-t/RC} \theta(t))\ .
$$
Now, since $s(t)$ is in units volts, I would suspect that the terms $1$ and $e^{-t/RC} \theta(t)$ are in units volts per time because $RC$ is in units time.
In another case, say, we want to know the impulse response function $h = h(t)$; i.e. the voltage $V_{out}$ that we measure over the capacitor if we hook up an impulse input voltage $V_{in}(t) = \delta(t)$, where $\delta$ is the Dirac delta function.
It can be shown that $h$ is given by the derivative of the step response $s$:
\begin{align}
h(t) &= \frac{d}{dt} s(t) \\
&= e^{-t/RC}\big(1-RC\delta(t)\big) \ .
\end{align}
$RC$ has units time. 
Both $s$ and $h$ are in units volts.
How on earth would I be able to use dimensional analysis in this case to check my results?
In the last expression $\delta(t)$ should now be in units volts per time for the equation to hold.
 A: Your understanding of the step response function is pretty flawed, though it's hard to pin down exactly how as you've not given a formal definition for what you're using.
To start with, the formula you've given is completely wrong, as it produces a discontinuous output, i.e. if you plot $(1-e^{-t}\theta(t))$ what you get is this:

This makes it pretty clear that the correct form for the response function should be
$$
s(t) = \mathrm{const}\times (1-e^{-t/RC})\theta(t),
$$
with the theta function as a global multiplier to make everything vanish before the voltage ramps up.
Now, as to the value of the constant, that's really up to how you define it, but there's really no reasonable understanding under which that constant should equal $RC$. You say that "it can be shown" that that is the correct form, but you don't give any details that would enable us to help you pinpoint where the mistake is inside your derivation, but it's simply not a workable relation.
Instead, you have two different options for the normalization of the step response function


*

*One is to have a dimensional forcing, $V(t) = V_0 \theta(t)$, with $V_0$ a constant with dimensions of voltage, in which case the response function is
$$
s_\mathrm{dim} (t) = V_0 (1-e^{-t/RC})\theta(t)
$$
and has the dimensions of voltage.

*The other one is to have a dimensionless forcing, $V(t) = \theta(t)$, with a unit (and therefore dimensionless) driving term at $t>0$, which will produce a response function of the form
$$
s_\mathrm{d.less} (t) = (1-e^{-t/RC})\theta(t)
$$
which, like the driving, is dimensionless.
Generally speaking, since the response function is best thought of as a property of the system, to be used in a convolution (or other similar mathematical device) with the actual driving voltage applied to the system, it is normally better to use the second understanding, and to include the units of volts on the actual physical driver and the actual physical response, while leaving the system's response unit-neutral.

Your second question is a bit trickier, because there the natural "unit-neutral" driver isn't actually dimensionless: that is,
$$
V(t) = \frac{\mathrm d}{\mathrm dt} \theta(t) = \delta(t)
$$
has dimensions of $[V(t)] = [1/T]$, i.e. inverse time. As you correctly point out, the impulse response function is given by the derivative of the step response function, so therefore it reads
\begin{align}
s_\mathrm{impulse} (t) 
& = \frac{\mathrm d}{\mathrm dt} s_\mathrm{step} (t) 
\\ & = \frac{\mathrm d}{\mathrm dt} (1-e^{-t/RC})\theta(t)
\\ & =  (1-e^{-t/RC})\delta(t) + \frac{1}{RC}e^{-t/RC}\theta(t)
\\ & =  \frac{1}{RC}e^{-t/RC}\theta(t),
\end{align}
where the delta-function term vanishes because it only cares about the behaviour at $t=0$, i.e. it pulls in the behaviour
$$
f(t) \delta(t) = f(0)\delta(t),
$$
which is obvious from the definition of the delta function, and in this case $f(t) = 1-e^{-t/RC}$ has $f(0)=0$.
Dimensionally, here both $e^{-t/RC}$ and $\theta(t)$ are dimensionless, giving $s(t)$ the dimensions of $[s(t)] = [1/RC] = [1/RC]$, i.e. an inverse time, which, as it should, matches the driver that produces it.

And finally, just to emphasize this: the number $1$ is always dimensionless, as is $\theta(t)$. If your formalism indicates that that's not the case, then you're doing things wrong: there's no such thing as "one, but with units of volts", there's only "you had a constant $V_0 = 1\:\rm V$ and you were sloppy with your constants", which you should never do.
A: If $f(x)$ has units of $X$, and $x$ units of $Y$, then $\frac{\delta f}{ \delta x}$ will have units of $X/Y$.
There seems to be bit of a confusion concerning $s(t)$ in your example. The characteristic equation of the system is $s(t) = -(RC)^{-1}$, and therefore, $s(t)$ is in units of time.
There is a nice tutorial explaining how to obtain the voltage step-response of the system here.
