It's actually not too hard to calculate the moment of inertia (MOI) of a right triangle. And you can make your triangle out of a large right triangle minus a smaller right triangle. So your MOI is just the MOI of the bigger triangle minus the MOI of the smaller one.
Step 1:
Extend line $b$ (move vertex $C$) until you have a right triangle. We'll calculate the MOI relative to vertex $A$. I agree with @fibonatic that it's easiest to use polar coordinates. So we have
\begin{equation}
I_{ABC'} = \int_0^\alpha \int_0^{R(\theta)} (\rho_A\, r^2) r\, dr\, d\theta~.
\end{equation}
Here, $\alpha$ is the angle $\angle B A C$, which you can calculate using trig. Also, $R(\theta)$ is the length of a line $Aa$ that goes from vertex $A$ to line $BC=a$, where $\theta$ is the angle of that line above line $b$. A little simple trig says that $R(\theta) = b/\cos\theta$.
\begin{align}
I_{ABC'} &= \rho_A\, \int_0^\alpha \int_0^{b/\cos\theta} r^3\, dr\, d\theta
\\
&= \rho_A\, \int_0^\alpha \left. \frac{r^4}{4} \right|_0^{b/ \cos\theta}\, d\theta
\\
&= \frac{\rho_A\, b^4}{4}\, \int_0^\alpha \cos^{-4}\theta\, d\theta
\\
&= \frac{\rho_A\, b^4}{24}\, \frac{3 \sin\alpha+\sin (3 \alpha
)} {\cos^3\alpha}
\end{align}
[I'm terrible at integrals, so I just looked up the answer for $\cos^{-4}$. But I think it's easy enough to convince you that you can handle any reasonable MOI problem.]
Step 2:
Use this formula and the parallel-axis theorem to get the MOI of the smaller right triangle that you need to remove from the bigger one to get your triangle.
Step 3:
Subtract the result of step 2 from step 1 to get the MOI of your triangle about the vertex $A$.
Step 4:
Use the parallel-axis theorem to get your MOI relative to whatever origin you want.
:(
.I have edited. $\endgroup$ – ABC May 16 '13 at 7:49