No, there is not a solar eclipse whenever we see a new moon. The reason we do not have a solar eclipse at every new moon is mostly due to the angle of Earth's axis (and by extension, the Moon's orbital plane) to the Earth-Sun line. See the picture below for a visual explanation. In the picture, the Sun is to the left. The upper image shows the orbit of the Moon during winter, the lower image shows the possible regions affected by the Moon at different times of the year.
At different times of the year, the Moon's orbit (which is more or less right above the equator) is tilted with respect to a line between the Sun and Earth. Only in the right conditions will you see a solar or lunar eclipse.
Additionally, it is worth noting that (and this is strictly technical) no New Moon that is not a solar eclipse (i.e. the observer is in the umbra) is a "full" New Moon. However, the sliver of light that is "visible" from Earth for these situations is too narrow and/or dim to be discerned by the human eye.
Image taken from http://www.uwgb.edu/dutchs/AstronNotes/HowSolSysWorks.HTM
For more information, see this similar question: Why is a new moon not the same as a solar eclipse?