Why triangular sails make a boat to sail upwind? I have read that triangular sails are good for upwind sailing whereas square sails are not.
What is so special about the triangular  shape that helps when sailing upwind?
 A: The issue is not the shape of the sail. It is the shape of the airfoil.
On a sailing yacht the mast forms the rounded leading edge, while the fabric forms the sharp trailing edge. That, plus angle of attack, is what makes a wing work.
Also on a single-hull sailing yacht, you have to consider the heeling moment. The boat has to "stand up" in the wind. If the "wing" had a larger "chord" high up on the mast, it would create a larger torque, tending to heel the boat over - not good.
Another issue is the out-haul - rope attached to the lower rear corner of the sail, needed to stretch it out. If the sail were rectangular it would need to be pulled at both top and bottom corners; for that there would need to be something like an upper spar.
I'm not so sure about racing catamarans, designed to "stand up" better. I think some of these actually have sails designed more like a wing and have greater chord near the top.
Of course, they have another issue to consider - minimizing the "wingtip vortex" at the top of the sail - which saps energy.
