Why trimming the mainsail gives a larger torque to head up I am learning sailing on a 5m catamaran (Nacra 5).
I am familiar with basic aerodynamics and the physics of the sail and keel.
We learned that when sailing closed hauled, too tight a mainsail tends to bring the boat up to the wind.  And that the opposite is true for the jib. For example, one may steer up to the wind to come about, using the mainsail alone, by trimming it tight.
My question is why the trimmed mainsail in the above setup gives a larger torque.
This also seems opposed to the rule: "The more the mainsail is sheeted out the more the boat tends to come up.", as explained in http://www.sailtheory.com/mandf.html#sailsteering
Edit:
To my understanding, there are several possible competing effects involved:
(1) Effects that tend to INCREASE the mainsail torque to head up:
(1A) The direction of the sail force becomes more perpendicular to the boat. This increases the heeling torque. Since heeling motion happens faster than turning, the boat will heel more. This moves the sail force out, which increases the lever arm and the torque to head up. see diagram in: http://www.sailtheory.com/mandf.html#hellingstuur
(1B) The sail force moves backwards since the sail is stretched backwards. This increases the lever arm and the torque.
(2) Effects that tend to DECREASE the mainsail torque to head up:
(2A) The direction of the sail force becomes more perpendicular to the boat. Assuming that the force is perpendicular to the sail, and that the center of rotation is between the mast and the center of force, one sees that pulling the sail in, reduces the torque to head up, as can be seen in the following diagram:  

As Theta gets smaller, the torque Tau is reduced. This is opposed to what is stated in some of the answers below.
(2B) The sail force is reduced since the sail was pulled beyond its optimal angle of attack, thus losing lift and reducing torque.
(2C) Due to (2B) heeling torque is also decreased. With a similar reasoning to (1A) this decreases the lever arm and torque.
(2D) The sail force moves forward since the aft part of the sail is too flat and stalls. This shortens the lever arm and reduces torque. This is described in http://www.sailtheory.com/mandf.html#sailsteering
We know from reality that 1A+1B > 2A+2B+2C+2D. Considering heeling alone, we also know that 1A > 2C. It remains to be explained why in general this is the case.
 A: A lot of what I begin with here will already be known by the asker, but just to give some background for my answer...
The action of the wind on the sails can be decomposed into two components: One which acts to move the boat forwards, and another which acts to push it sideways. The sideways force is countered by the keel/centerboard/daggerboard, which leaves the forward force to propel the boat. For this discussion I'll ignore the resulting torque which acts to heel the boat over.
The force of the water on the keel can be treated as if it acts laterally at one position along the length of the boat, and likewise for the lateral forces from each sail. These lateral forces each produce a torque that would tend to turn the boat, and they are not, in general, going to balance each other. Once the sails are trimmed for a given angle of attack to the wind, this residual torque is countered by the rudder. In this way the boat reaches a steady state where all torques are in equilibrium and it is propelled in the desired direction.
If the sail trim is then adjusted, the force from that sail will change in both magnitude and direction, both of which will alter the lateral component of force from that sail. The overall force from a sail is presumably in a direction close to (but I'd guess not exactly) perpendicular to the sail surface. Therefore, when the sail is pulled in close, more of its force is directed laterally. With all other trim being held constant, this change in torque would begin to turn the boat.
If the excess force is acting at the rear of the boat (i.e. the mainsail is pulled in), then the rear of the boat would be pushed upwind, which would obviously mean that the boat is turning in to the wind. Similarly, if the jib is pulled in, the extra lateral force would push the bow away from the wind.
As other answers have said, this likely violates your rule: "The more the mainsail is sheeted out the more the boat tends to come up." because that rule is given in the context of a different rig. In a single-sail rig, for example, the center of lateral force from the mainsail may be well forward of the center of lateral force from the centerboard, while on your catamaran, the daggerboard appears to be even with the mast, which would put the center of lateral force from the mainsail well behind the daggerboard.
A good illustration of this effect is the Yawl rig. The small mizzen doesn't provide much propulsion, but it's position at the extreme read of the boat allows it to provide fine adjustment of the torque on the boat. This is what allows the yawl to be trimmed to sail without a hand on the tiller.
A: There are two main reasons why close hauling brings your bow to the wind.
One reason is that the trimmed mainsail gives a larger torque: The equivalent force in a freebody diagram is larger for a trimmed (close hauled) mainsail.  Note that the acutal force does not have to change in order to accomplish this, since |torque| = |F X L|; where F = force, L = length of lever arm, both are vector quantities, and 'X' is the vector cross product operator.
Consider the main mast as the pivot point for the boat (this is almost certainly not exactly true, but consider it so for the moment).  The lever arm which the mainsail uses to pivot the boat about this point is the distance from the mainsail mast to the belaying point or winch for the line which stays the mainsail in whatever position you want it.  This length does not change.
So, L does not change.  F does not change.  What changes?  The angle between the two vectors.  When F and L are at almost 90 degree angles, "F x L" is at its largest.  This occurs when close hauled. 
A way to visualize the equivalent torque is to picture the orientation of the boat and mast from above.  Assume that the mainsail cross member is as long as the distance from the mast to the belaying point.  Assume also that the mainsail is let out to where it forms a 90 degree angle with the keel line.  That means the mainsail belaying line will form a 45 degree angle with the keel line and the mainsail.  Lets say that 'F' is 1000 lbs.  That tension can be broken into a force and a torque. sine(45) = cosine(45) = 1/1.414 = 0.707.  So the boat experiences 1000sin(45) = 707 lbs of forward thrust through the water.  This force is being aplied to the boat at the belaying point of the mainsail line.  The boast also experiences 1000cos(45) = 707 lbs. of force as the 'F', in a pure (F)(L) torque.  
Now lets close haul:  'F' and 'L' are the same, and the mainsail makes a 30 degree angle with the keel line.  The forward thrust to the boat is 1000sin(30) = 500 lbs.  The 'F' in the torque component is 1000cos(30) = 866 lbs.  That is almost a 24% increase in 'F', which will result in a 24% increase in torque (since L remains constant).
The second reason that close hauling brings the bow into the wind is to think of the mainsail as a weathervane, with the wind coming from just a few degrees aft of a ray which is co-axial with the mainsail cross arm and pointing forward through the mast.  This never changes (significantly, in the steady state, and assumes you are not "running before the wind").  So, on any given tack, the position of your mainsail relative to the wind does not change.  Given this, what will happen if you haul the mainsail closer to your boat?  Relative to the wind, you are not really hauling the mainsail anywhere, since it stays fixed relative to the wind.  What you are really doing is hauling your keel line (boat's axis) closer to the mainsail's axis, which brings the bow closer to the wind.  
A: From my sailing experience (which does not include catamarans specifically) a tight sail always wants to turn the boat up into the wind. It is the partial cancellation of this force with the stability provided by a keel and/or centerboard that allows you to move forward at a close haul. Because the force on the boat from the sail always acts at 90 degrees from wherever the boom line is, pulling the sail in close creates a stronger torque that needs to be canceled if you want your boat to stay right side up. Usually this means using your body weight as a counter balance. 
Forcing the jib out can increase the air flow around the foil changing the apparent wind direction, though I'm not sure if this is what you are talking about..
