It's not quite clear what the inner workings of your function are, because your $K$ does not match 't Hooft's, and you have not provided your working. However, there is a simpler test case which encapsulates the problem better.
Consider the zero-order modified Bessel function $K_0(x)$, which has the integral representation
$$\mathop{K_{0}}\left(x\right)
=
\int_{0}^{\infty}{\cos}
\left(x{\sinh} t\right)\mathrm dt
\tag1
$$
as an equivalent to yours. If you plug it into the modified Bessel equation
$$
\left[x^{2}\frac{{d}^{2}}{{dx}^{2}}+x\frac{d}{dx}-(x^{2}+\nu^{2})\right]K_\nu(x)=0,
$$
and do your integration by parts, you're left instead with the residue
$$
\left[x^{2}\frac{{d}^{2}}{{dx}^{2}}+x\frac{d}{dx}-(x^{2}+\nu^{2})\right]K_\nu(x)
=
-\lim_{t\to\infty}x\cosh(t)\sin(x\sinh(t)),
$$
which diverges to infinity (with some wild mood (erm, sign) swings along the way).
This modified case is simpler do deal with, and the solutions are obviously the same. The problem, in both cases, is that the differentiation and the improper integral do not commute and, indeed, that the differentiated integrals don't make sense at all.
I'll show this for the case of $K_0(x)$. This is easier to do with the simpler representation
$$
K_0(x)
=\int_0^\infty\cos(xu)\frac{\mathrm du}{\sqrt{1+u^2}}
=\lim_{U\to\infty}\int_0^U\cos(xu)\frac{\mathrm du}{\sqrt{1+u^2}},
$$
where a change of variable has traded the cancellations of a constant-amplitude integrand in $(1)$ for a decreasing integrand. However, the integrand does not decrease fast enough (i.e. it goes as $1/u$ at large $u$), and the convergence of this integral still depends on cancellations.
Differentiating under the integral sign amounts to commuting the limit with the derivative:
$$
\frac{d}{dx}\lim_{U\to\infty}\int_0^U\cos(xu)\frac{\mathrm du}{\sqrt{1+u^2}}
\stackrel{?}{=}
\lim_{U\to\infty}\frac{d}{dx}\int_0^U\cos(xu)\frac{\mathrm du}{\sqrt{1+u^2}}
.
$$
However, if you actually do the integral on the right,
$$
\frac{d}{dx}\lim_{U\to\infty}\int_0^U\cos(xu)\frac{\mathrm du}{\sqrt{1+u^2}}
\stackrel{?}{=}
-\lim_{U\to\infty}\int_0^U\sin(xu)\frac{u}{\sqrt{1+u^2}}\mathrm du
,
$$
you get a result that can't be handled via the usual theory of functions. The factor $\tfrac{u}{\sqrt{1+u^2}}$ does not go to zero (it tends to $1$ from below), and the integral will not converge at all. In fact, if you re-write it as
$$
\int_0^U\sin(xu)\frac{u}{\sqrt{1+u^2}}\mathrm du
=
\int_0^U\sin(xu)\left[1-\frac{1}{\sqrt{1+u^2}(\sqrt{1+u^2}+u)}\right]\mathrm du
$$
then you get a perfectly well-behaved (i.e. uniformly and absolutely convergent) term on the right, plus the integral $\int_0^U\sin(xu)\mathrm du$, which can only be consistently handled via distribution-theoretic methods. (Another term to google: "generalized function".)
If you want do do this right, then, you need to see equations like
$$
\frac{d}{dx}K_0(x)=-\int_0^\infty\sinh(t)\sin(x\sinh(t))\mathrm dt
$$
as holding only in a suitable distributional sense, though after your calculation is done you can bring the thing back down to the domain of everyday functions.
By "suitable distributional sense" I mean the same sort of techniques as used to make sense of the integral
$$
\int_{-\infty}^\infty e^{ixt}\mathrm dt,
$$
which you should recognize as being $2\pi\delta(x)$, the Dirac delta function. The problem, of course, is that the improper integral makes no sense, and does not converge. To deal with this sort of thing in a rigorous way, we say that
$$
\int_{-\infty}^\infty e^{ixt}\mathrm dt
\stackrel{\mathrm{DS}}{=}
2\pi\delta(x),
$$
where the $\mathrm{DS}$ is read out as "in the distributional sense", and the equation actually means that
$$
\int_{-\infty}^\infty\int_{-\infty}^\infty\Phi(x)e^{ixt}\mathrm dx\mathrm dt
=2\pi\int_{-\infty}^\infty\Phi(x)\delta(x)\mathrm dx
\quad\text{for all }\Phi\in\mathcal S.
$$
Here we take the $x$ integral before the improper $t$ one. The function $\Phi$ is called a test function and you require it to be as nice as necessary (typically $C^\infty$ and with bounded support).
Similarly, you can define
$$
K_0(x,U)
=\int_0^U\cos(xu)\frac{\mathrm du}{\sqrt{1+u^2}},
$$
for which
$$
\frac{d}{dx}K_0(x,U)
=
-\int_0^U\sin(xu)\left[1-\frac{1}{\sqrt{1+u^2}(\sqrt{1+u^2}+u)}\right]\mathrm du.
$$
Here you want to take the limit of $U\to \infty$, but the thing on the right won't converge. What you need to do, then, is integrate against a test function:
$$
\int_{-\infty}^\infty\Phi(x)
\frac{d}{dx}K_0(x,U)
\mathrm dx
=
\int_{-\infty}^\infty\Phi(x)
\int_0^U\sin(xu)\left[1-\frac{1}{\sqrt{1+u^2}(\sqrt{1+u^2}+u)}\right]\mathrm du
\mathrm dx
.
$$
The cool thing here is that if $\Phi$ is nice enough, then you can force the ugly term to vanish. In particular, take $\Phi$ to be smooth and with finite support (so the $x$ integral has finite limits in every case), so you can do
\begin{align}
\int_{-\infty}^\infty\Phi(x)
\int_0^U\sin(xu)\mathrm du
\mathrm dx
&=
\int_{-M_\Phi}^{M_\Phi}\Phi(x)
(1-\cos(xU))
\mathrm dx
\\&=
2
\int_{-M_\Phi}^{M_\Phi}\Phi(x)
\sin(2xU)
\mathrm dx.
\end{align}
This is the integral of a smooth function multiplied by a factor that oscillates at higher and higher frequencies. This is governed by the Riemann-Lebesgue lemma, which says the whole thing goes to zero at $U\to\infty$. This means that in the limit, you can drop the ugly term:
$$
\lim_{U\to\infty}
\int_{-\infty}^\infty\Phi(x)
\frac{d}{dx}K_0(x,U)
\mathrm dx
=
\lim_{U\to\infty}
\int_{-\infty}^\infty\Phi(x)
\int_0^U\frac{\sin(xu)\mathrm du}{\sqrt{1+u^2}(\sqrt{1+u^2}+u)}
\mathrm dx
.
$$
This is very clunky, so we usually abbreviate it by saying
$$
\frac{d}{dx}K_0(x)
=
\lim_{U\to\infty}
\frac{d}{dx}K_0(x,U)
\stackrel{\mathrm{DS}}{=}
\lim_{U\to\infty}
\int_0^U\frac{\sin(xu)\mathrm du}{\sqrt{1+u^2}(\sqrt{1+u^2}+u)},
$$
i.e. the improper integral on the right converges in the distributional sense (only) to the intermediary term in the middle. This limit, on the other hand, is perfectly well defined and equals the normal Bessel function term on the left.
And so you go. You keep on working on these terms, where some functions are equal in the regular sense and some are only equal in the distributional sense to ugly integrals that don't actually converge. By the end, though, you'll get some combination of derivatives equal to zero in the distributional sense, but if you choose your class of test functions to be broad enough, then the only way that can happen is if the differential equation holds in the regular sense, with no monkey business with distributions.
