As to your first question, why does the particles being more tightly bound decrease rather than increase their rest mass, think of it this way: when the nucleons are more tightly bound, you need to add energy to the system to force them farther apart. In other words, you need to do work (i.e. expend energy) to pull the nucleons further apart from each other. When you add energy, you add (relativistic) mass.
As to your second question, of course the mass difference for a single atomic reaction is miniscule on macroscopic scales. However, consider the difference between the energy released per fusion reaction (17.6 MeV in the case of DT fusion) and the energy released in a chemical reaction (e.g. burning coal), which is on the order of 1-10 eV per reaction (depending on the substance you are burning etc.). From this you can see that the difference in energy released per reaction is six orders of magnitude - nuclear (fusion, but also fission) releases a million times more energy per reaction than burning something. Scale this up to a fusion reactor where the fuel density is on the order of $10^{20}$ particles/m$^{3}$ in a reactor of several (hundreds of) cubic meters in volume, and you get a very large energy output.
EDIT: in response to Hardik's additional question:
There are forces at work here, the strong nuclear force and the electromagnetic force. The way the nucleons are bound are through the strong nuclear force, which easily overcomes the electromagnetic repulsion of the positive protons in the nucleus but only at short distances. So before you can get the reacting nuclei close enough together so that the strong force takes over and binds the nucleons into a new, more tightly bound and therefore lighter particle, you need to do a lot of work to overcome the electromagnetic repulsion. This is why fusion plasmas need to be so hot; the particles need enough kinetic energy to overcome this energy barrier that the electromagnetic force creates. However, once you overcome this barrier, the potential well on the other side, where the strong force takes over, is much deeper than the energy you had to put in to bring the particles close. This is illustrated nicely in this schematic:
In this figure, imagine one particle is sitting at the origin on the left, while the other particle approaches from the right. You need to put in an amount of energy equal to the shaded purple area to get the right particle over the Coulomb barrier, but then it falls into the strong force potential well which releases the shaded green area of energy, which is greater than the shaded purple area because the well is so deep (the strong force is so much stronger than the electromagnetic force). This is the energy that is released in a fusion reaction.
EDIT 2: In response to a second additional question: exactly, helium is more tightly bound than deuterium and tritium and the difference is released as energy. This 17.6 MeV is not again going into binding energy because quantum mechanics dictates the binding energies of the reacting particles and the reaction products, so those are fixed. The leftover energy then has to go somewhere else. The I'm not sure exactly what you mean by "free" energy, but this 17.6 MeV of energy is released as kinetic energy of the reacting products. There's simply nowhere else for that energy to go. In other words, the helium and neutron that are generated are given a big "kick" of kinetic energy when they are created in the reaction. That kinetic energy is distributed according to conservation of momentum, so 3.5 MeV to the alpha particle (helium nucleus) and 14.1 MeV to the neutron. If you're wondering why that specific number, it's because the reaction products contain 5 total nucleons (3 neutrons and 2 protons, that have approximately the same mass). The alpha particle contains 4/5th of the nucleons and the neutron has 1/5th, so the energy is distributed so that 1/5*(17.6 MeV) = 3.5 MeV goes to the alpha particle and 4/5*(17.6 MeV) = 14.1 MeV goes to the neutron.
I hope this answers your question(s).