If i have a small toy boat with a sail, and i attach a fan onto it, FACING THE SAIL, which runs by solar. Once the fan turns on will the boat move or will it remain at rest. Apparently it wont because according to Newtons third law the sail will give an equal force back on to the air pushed by the fan. But that doesnt really make sense, so how do real boats which rely on the wind work? If I turn the fan the other way round, the boat will move as the model is now like a planes engine but why not when the fan is facing the sail??
I think you are mixing fan and sail effects.
If you have fan only, then fan is pushing air backward and so air is pushing boat forward.
If you have only sail, i.e. sailing in nature, then moving air (wind) pushes boat forward, and yes, boat is pushing air backward, but since mass of the whole moving air is huge it has negligible effect on it.
If you have both then suppose fan is in the front of the boat and sail in the back. Fan is pushing air backward, and air boat forward, but the opposite effect happens at sail: air pushes boat back and sail pushes air forward. The two effect cancel each other.
fan <--> air <--> sail
With a sail and a fan it will not move as you rightly argued from a balance of forces. Even turning the fan around will not lead to much movement, as the fan is pulling on the sail (but to a lesser degree).
In real sail boats you can imagine the fan being on the shore and now the boat can move, as there is no fixed connection between the boat and the fan anymore.