The author wants to write potential energy of the ball attached to pendulum. He says that pivot is at $y=0$, so $y$ component is $-\ell\cos\vartheta$ so $U=-mg\ell\cos\vartheta$.
But why don't we write $-mg$ as a force which should give us $-mg\times-\ell\cos\vartheta$. The minus sign for $mg$ comes from the fact that $y$ direction is upwards. If below $y=0$, it is considered negative. Any ideas?
One thing we must remember is that the potential energy of an object doesn't have any significance. The only thing that matters is the difference in potential energies between two points on the trajectory. That said, we know that this difference is given by $U(b) - U(a) = -\int_{a}^{b}\vec{F} \cdot \vec{dr}$ , where a and b are the initial and final points, $\vec{F}$ is a conservative force. Now, in your case, if we take downward direction as -ve, then $\vec{F} = -mg\hat{y}$ and the $\vec{dr} = -ld\theta(cos\theta\hat{x} + sin\theta\hat{y})$. Dot product gives $mglsin\theta d\theta$. This is to be integrated from $90^{\circ}$, the initial position to $\theta$, the final position. This gives $mglcos\theta.$ But beware! We haven't included the minus sign from the definition of potential energy, which gives the right answer.
Edit: Okay I just realised I made a slight mistake: integral of $sin\theta$ is $-cos\theta$, which will lead to the erroneous result you have obtained. The reason is that when we formulate potential, we always talk about it as the work done by an external force in causing the displacement, WITHOUT causing an acceleration to the object, and moving against the force field. Here, it is all about looking at the same displacement, but in order that the acceleration be zero, the force must be $\vec F = mg\hat y$. On a final note, the formula for potential energy difference is $U(b) - U(a) = \int_{a}^{b}\vec{F_E} \cdot\vec{dr}$ where all the variables are the same as before, except that the subscript E denotes the external force required, which is negative of gravitational force, and $\vec{dr} = ld\theta(cos\theta)\hat{x} + ld\theta(sin\theta)\hat{y}$, as this is the displacement against the force field. Setting $a = \theta$ and $b = \pi/2$ and using the assumption that $U(\pi/2)=0$ gives the answer. :)
The formula you're using, force times distance, tells you how much work the force does on the object in question. In other words, it correctly calculates how much work gravity does on your ball if it were to fall from horizontal to vertical. So it's how much kinetic energy the ball would have if if fell from the horizontal position ($\theta=90\text{ deg}$) to the vertical position ($\theta=0\text{ deg}$).
Potential energy plus kinetic energy is conserved, so that gain in kinetic energy comes with a loss of potential energy. You were missing a minus sign if you intended to calculate potential energy. So when writing the potential energy you should say $(-1)(-mg)(-l\cos\theta)$.
The torque about the pivot is:
$$\vec \tau=\vec P\times\vec F$$
where $~\vec P~$ is the position vector to the mass and $~\vec F~$ is the gravitation force
with
$$\vec P=\left[ \begin {array}{c} l\sin \left( \vartheta \right) \\ -l\cos \left( \vartheta \right) \\ 0\end {array} \right] \quad, \vec F=\left[ \begin {array}{c} 0\\ -mg \\ 0\end {array} \right] $$
you obtain
$$\vec \tau= \left[ \begin {array}{c} 0\\ 0\\ -l\sin \left( \vartheta \right) mg\end {array} \right] $$
which is correct , the torque about the pivot is negative due to the negative gravitation $~g~$
