Solution using a basis set expansion
Let $H_0|n\rangle=E_n|n\rangle$ and lets make the following ansatz for our time dependent state,
$$
|\psi(t)\rangle = \sum_n c_n(t) \exp\left(-\frac{i}{\hbar }E_nt\right)|n\rangle .
$$
Inserting this into the time dependent Schrödinger equation $i\hbar \frac{d}{dt}|\psi(t)\rangle = H(t)|\psi(t)\rangle$ with $H(t)=H_0 +\delta(t)V$ leads after cancelation of some terms to
$$
i\hbar \sum_n \dot c_n(t) \exp\left(-\frac{i}{\hbar }E_nt\right)|n\rangle = \sum_l c_l(t) \exp\left(-\frac{i}{\hbar }E_lt\right)V\delta (t)|l\rangle
$$
Projection unto state $\langle k|$ yields with the definition $V_{kl}= \langle k|V|l\rangle$ and using the orthonormality of states $\langle k|n\rangle = \delta_{kn} $
$$
i\hbar \dot c_k(t) \exp\left(-\frac{i}{\hbar }E_kt\right)= \sum_l V_{kl} c_l(t) \exp\left(-\frac{i}{\hbar }E_lt\right)\delta(t)
$$
Now we multiply with $$-\frac{i}{\hbar}\exp\left(\frac{i}{\hbar }E_kt\right) $$ and define $\tilde V_{kl}(t)=V_{kl}\exp\left(-\frac{i}{\hbar }(E_l-E_k)t\right)$ which leads to
$$
\dot c_k = -\frac{i}{\hbar } \sum_l \tilde V_{kl}(t) c_l(t)\delta(t)
$$
This can be written as vector equation
$$
\mathbf{ \dot c} = -\frac{i}{\hbar } \mathbf{\tilde V(t)c(t)}\delta(t)
$$
We can now integrate the equation,
$$\begin{aligned}
\int_0^t dt' \mathbf{ \dot c} &= -\frac{i}{\hbar } \int^t_0 dt' \mathbf{\tilde V(t')c(t')}\delta(t')\\
\mathbf{c(t)} - \mathbf{c(0)} &= -\frac{i}{\hbar } \mathbf{\tilde V(0)c(0)}\\
\mathbf{c(t)} &= \left(\mathbf 1 -\frac{i}{\hbar } \mathbf{\tilde V(0)}\right )\mathbf{c(0)}
\end{aligned}$$
This equations holds for $t \geq 0$. If we assume that our system propagates freely before $t=0$, we can introduce the Heaviside theta function to obtain
$$
\mathbf{c}(t) = \left(\mathbf 1 -\frac{i}{\hbar } \mathbf{\tilde V(0)}\theta(t)\right )\mathbf{c(0)}
$$
Solution without basis set expansion
We could also do the same without expansion in an explicit basis(which is just a bad habit of mine due to my Comp. Chem. background ...)
For that define the state in the interaction picture as
$$\begin{aligned}
|\psi(t)\rangle = \exp\left({-\frac{i}{\hbar}}H_0 t\right) |\psi_I(t)\rangle \\
|\psi(0)\rangle = |\psi_I(0)\rangle
\end{aligned}$$
Now insert this into the TDSE,
$$\begin{aligned}
i\hbar \frac{d}{dt}|\psi(t)\rangle &= (H_0 + V\delta(t) )|\psi(t)\rangle \\
i\hbar \frac{d}{dt} \left(\exp\left({-\frac{i}{\hbar}}H_0 t\right) |\psi_I(t)\rangle \right) &= (H_0 + V\delta(t) )\left(\exp\left({-\frac{i}{\hbar}}H_0 t\right) |\psi_I(t)\rangle \right) \\
i\hbar \left( -\frac{i}{\hbar}H_0 \exp\left({-\frac{i}{\hbar}}H_0 t\right) |\psi_I(t)\rangle +\exp\left({-\frac{i}{\hbar}}H_0 t\right) |\frac{d}{dt}\psi_I(t)\rangle \right) &= (H_0 + V\delta(t) )\left(\exp\left({-\frac{i}{\hbar}}H_0 t\right) |\psi_I(t)\rangle \right)\\
H_0\left(\exp\left({-\frac{i}{\hbar}}H_0 t\right)|\psi_I(t)\rangle \right)+i\hbar\left(\exp\left({-\frac{i}{\hbar}}H_0 t\right) |\frac{d}{dt}\psi_I(t)\rangle \right) &=(H_0 + V\delta(t) )\left(\exp\left({-\frac{i}{\hbar}}H_0 t\right) |\psi_I(t)\rangle \right)\\
\text{Cancel }H_0 \text{ term on both sides}\\
i\hbar\left(\exp\left({-\frac{i}{\hbar}}H_0 t\right) |\frac{d}{dt}\psi_I(t)\rangle \right) &= V\delta(t)\left(\exp\left({-\frac{i}{\hbar}}H_0 t\right) |\psi_I(t)\rangle \right)\\
\text{Multiply with }\exp\left(\frac{i}{\hbar}H_0 t\right)\text{ from the left}\\
i\hbar |\frac{d}{dt}\psi_I(t)\rangle &= \exp\left(\frac{i}{\hbar}H_0 t\right)V\delta(t)\exp\left({-\frac{i}{\hbar}}H_0 t\right) |\psi_I(t)\rangle \\
\int^t_0 dt' |\frac{d}{dt'}\psi_I(t')\rangle & =-\frac{i}{\hbar} \int^t_0 dt' \exp\left(\frac{i}{\hbar}H_0 t'\right)V\delta(t')\exp\left({-\frac{i}{\hbar}}H_0 t'\right) |\psi_I(t')\rangle \\
|\psi_I(t)\rangle - |\psi_I(0)\rangle &= -\frac{i}{\hbar} \exp\left(\frac{i}{\hbar}H_0 \cdot 0\right)V\exp\left({-\frac{i}{\hbar}}H_0 \cdot 0\right) |\psi_I(0)\rangle \\
|\psi_I(t)\rangle &= |\psi_I(0)\rangle -\frac{i}{\hbar}V|\psi_I(0)\rangle\\
|\psi_I(t)\rangle &= (\mathbf{1} -\frac{i}{\hbar}V)|\psi_I(0)\rangle\\
\exp\left({\frac{i}{\hbar}}H_0 t\right) |\psi(t)\rangle &= (\mathbf{1} -\frac{i}{\hbar}V)|\psi(0)\rangle\\
|\psi(t)\rangle &=\exp\left(-{\frac{i}{\hbar}}H_0 t\right)(\mathbf{1} -\frac{i}{\hbar}V)|\psi(0)\rangle\\
\end{aligned}$$
This is equivalent to my first calculation. The line
$$
\mathbf{c}(t) = \left(\mathbf 1 -\frac{i}{\hbar } \mathbf{\tilde V(0)}\right )\mathbf{c(0)}
$$
is equivalent to
$$
|\psi_I(t)\rangle = (\mathbf{1} -\frac{i}{\hbar}V)|\psi_I(0)\rangle\\
$$