General scattering theory I have studied a bit of scattering/diffusion theory in an introductory course in quantum mechanics (the kind where the scattering potential is delta, box, and similar easy functions). But when the potential is everywhere continuous (asymptotically vanishing let's say in $\pm \infty$), what kind of tools can we use to study the transmission and reflection coefficients? 
 A: You may be worried about the boundary conditions: the difference between the smooth condition or the discontinuous boundary condition.
In general, if you are thinking about the Transmission and Reflection coefficients, a powerful method is so called the WKB (Wentzel–Kramers–Brillouin) approximation.
For example, such a continuous potential is solved via WKB in Example 8.4 of Griffiths Intro to Quantum Mechanics (2nd edition).
In other case, you may not need to define the Transmission and Reflection coefficients, especially when the system is not in 1d, or with a symmetric 1d  line potential in 2d, or with a symmetric 2d plane potential in 3d.
In some case, such as in 3d space, it is convenient to study the differential cross section, which one can use the tools of (1) pertubative series, order by order expansion, via so called the Born approximations. Or, (2) the Partial Wave analysis, where one match the incoming wave $\psi_{in}$ such as a plane wave, with the outgoing wave $\psi_{out}$ expanded around the eigenfunctions based on the symmetry of the background potential.
A classic example for this is Coulomb interaction with a $S^2$ spherical rotational symmetry, thus SO(3) invariant. It is a continuous potential, and one can solve it via the two methods described above.  
You may also check the book Modern Quantum Mechanics by Sakurai.
