Mathematically, what is the kernel in path integral? The kernel in path integral to transform the wave function (Eq 3.42 in Feynman and Hibbs, Quantum Mechanics and Path Integrals, emended edition):
$$\psi(x_b,t_b)=\int_{-\infty}^\infty K(x_b,t_b;x_c,t_c)\psi(x_c,t_c)dx_c$$
looks equivalent to the kernel to transform the integral:
$$(Tf)(u) = \int \limits_{t_1}^{t_2} K(t, u)\, f(t)\, dt$$
However, the table of transforms doesn't have any "Feynman transform". What exactly is it in mathematics?
 A: For a given quantum system, the kernel of the path integral is, in fact, the kernel of an integral transform as you explicitly write down.  It is the transform that governs time evolution of the system as is manifest in your first equation.  For this reason, it is often referred to as the propagator of a given system.  
For example, for a single, non-relativistic particle moving on some portion of the real line with a constant Hamiltonian (and thus unitary evolution), the kernal and unitary time-evolution operator are related as follows:
\begin{align}
K(x,t; x', t') = \langle x|U(t,t')|x'\rangle.
\end{align}
Kernels for distinct quantum systems will, in general, be different, because these systems have different hamiltonians and thus different behaviors under time evolution.  Contrast this to the e.g. the Fourier and Laplace transforms whose kernels are always the same (up to pesky conventions of course).
For example, the kernel for a free massive, non-relativistic particle moving on the real line is
\begin{align}
  K(x_b t_b; x_a, t_a) = \left[\frac{2\pi i\hbar(t_b-t_a)}{m}\right]^{-1/2}\exp\frac{im(x_b-x_a)^2}{2\hbar(t_b-t_a)}
\end{align}
and you could look up the kernels of a number of other systems (like the harmonic oscillator).  
The main point. There is a different "Feynman transform," as you put it, for each quantum system.  This is why you won't find just one in a table of mathematical transforms under some such name.  This does bring up another interesting question though: is there a table of known kernels for various quantum systems somewhere?  I'd be interested to know myself!
More on specific kernels and some generalizations here: http://en.wikipedia.org/wiki/Propagator
