For what concerns waveguides, we are interested in transverse modes, i.e. a light wave whose transverse field distribution remains unchanged as it propagates. To be clear we are interested in the transverse field distribution $E(x,y)$ (or $H(x,y)$). Roughly speaking, this would be the pattern of the field you can see on a section perpendicular to the propagation direction of your light beam.
Finding waveguide modes means finding the solution of the Maxwell equations under the boundary conditions specified by the physical structure of your waveguide. So, depending on the structure of the waveguide, you will have different waveguide modes. A planar waveguide, a rectangular waveguide or an optical fiber have different modes because they have a different physical structure.
Usually we only concentrate on guided modes, which are solutions to the boundary problem whose field distribution is restricted mainly on the core region of the fiber. But guided modes are not the only solution to this problem. We have also an infinite continuum of so-called radiation modes, which are not guided but which correspond anyway to a solution to the same boundary problem.
Generally we are interested in guided modes and we assume an infinite cladding radius, where $n_1$ is the refractive index of the core and $n_2$ is the refractive index outside the core (in the infinite cladding).
The condition for guided modes is
n_2 k< \beta<n_1 k
where $\beta$ is the propagation constant. Guided modes vary harmonically in the core but decay exponentially outside of the core region. To add more detail, the so-called fundamental mode of the waveguide concentrates all its power in the core while higher-order modes penetrates a bit more outside the core. So far, so good.
For radiation modes we have
and light is allowed to "escape" out of the core. In the specific case of a step-index waveguide of infinite cladding radius we will find them oscillating inside and outside the core.
Here's a picture (from Sensors, volume 6 - Gopel, Hesse, Zemel) of guided and radiation modes of an asymmetric slab waveguide:
and you may see that different radiation modes have different behaviours, in one case varying harmonically over the whole range, in the other case decaying exponentially in the cladding ($n_c$ index) region.
Of course the infinite cladding approximation, even if it is useful for guided modes, is not a very good approximation to analyse radiating modes, as we have to take into account even the cladding/coating or cladding/air interface in a real situation.
For a finite cladding radius the concept of cladding modes arises. Cladding modes are modes that remain "trapped" into the cladding of the fiber, due to the fact that $n_3 k< \beta<n_2 k$, where $n_3$ is the refractive index of the coating medium or of air in the case of an uncoated fiber and that can propagate. Usually these modes are unwanted and we can get rid of them by using a coating which has higher refractive index of the cladding so that the condition of reflection at the cladding-coating interface will not be fulfilled: this part of light will radiate away and it will be lost. But I know that there are fibers (double clad fibers) which allow for the propagation of cladding modes. I don't know any details about these fibers though.
So, when you ask "But why they can't still propagate and transfer energy simply by reflection from two boundaries of the waveguide" I don't know what you mean with "boundaries". If you mean the cladding/coating interface then these modes, cladding modes, can exist and they can propagate and transfer energy, if we want them to. It's just that usually we don't want them to do that.
You can see here a picture (from Thorlabs) of a cladding mode:
Any field that we inject in the waveguide can be decomposed into guided and radiation modes. This means that, knowing which modes our input light is "made of", we can follow these modes independently along the waveguide. The input modes which correspond to the guided modes will be guided, the others will radiate away. For example, imagine that you have a single-mode fiber (a fiber that guides only ONE mode), which only guides the mode that we call "mode A". If your input light is very different from this mode (that is to say that if you decompose your input light only a small fraction of it is made of the mode "A") you will have a very bad transmission thoughout your fiber. If you want a good transmission you have to "modify" your input light so that it is the closest possible to the mode "A". So, for your second question, the answer is "it depends on your input beam".
I have a single-mode non-linear waveguide in my lab and I spent days working on mode-matching: I tried different lenses to adjust the waist of my beam at the input to match the correct "size" of the guided mode and I changed the numerical aperture as well. Again, the goal is to make your input beam the closest possible to the guided mode of the waveguide.