A polygon can be decomposed into multiple triangles. Suppose you want to find the MMOI value about a reference point (the coordinate origin) and about the center of mass. Split the poly gone into multiple triangles and follow the following algorithm:
Loop through triangles, each with three vertices $\boldsymbol A=(A_x,A_y)$, $\boldsymbol B=(B_x,B_y)$, $\boldsymbol C=(C_x,C_y)$
Calculated the area of the triangle $$ {\rm area}(i) = \tfrac{1}{2} \left( \boldsymbol{A} \times \boldsymbol{B} + \boldsymbol{B} \times \boldsymbol{C} + \boldsymbol{C} \times \boldsymbol{A} \right) \tag{1} $$
where $\boldsymbol{A}\times \boldsymbol{B} = A_x B_y - A_y B_x$ and so on with the remaining cross products.
Calculate the centroid of the triangle $$ {\rm cen}(i) = \tfrac{1}{3} \left( \boldsymbol{A} + \boldsymbol{B} + \boldsymbol{C} \right) \tag{2}$$
Calculate the mass moment if inertia of the triangle (per unit mass) $$ {\rm mmoi}(i) = \tfrac{1}{6} \left( \boldsymbol{A}\cdot\boldsymbol{A}+ \boldsymbol{B}\cdot\boldsymbol{B} + \boldsymbol{C}\cdot\boldsymbol{C} + \boldsymbol{A}\cdot \boldsymbol{B} + \boldsymbol{B} \cdot \boldsymbol{C} + \boldsymbol{C}\cdot \boldsymbol{A} \right) \tag{3}$$
where $\boldsymbol{A} \cdot \boldsymbol{B} = A_x B_x + A_y B_y$ and the same for the remaining dot products.
After finishing looping through the triangles, and given the total mass $m$ is known you calculate the following values for the polygon.
- Total area $${\rm AREA} = \sum_i {\rm area}(i)$$
- Density $$\rho = m / {\rm AREA}$$
- Center of mass $$\boldsymbol{G} = \frac{1}{\rm AREA} \sum_i {\rm area}(i)\, {\rm cen}(i)$$
- MMOI about origin $$I = \rho \sum_i {\rm area}(i)\, {\rm mmoi}(i)$$
- MMOI about the center of mass $$ I_G = I - m\,(\boldsymbol{G} \cdot \boldsymbol{G}) $$
Note: the 3D version of the above can be found in this post of mine that includes verification using CAD.
Appendix I
The development of the formulas goes as follows. The interior of the triangle is parametrized with $u=0\ldots 1$ and $v=0\ldots1$ as follows $$ {\rm pos}(u,v) = (1-u) \boldsymbol{A} + u (1-v) \boldsymbol{B} + u\,v\,\boldsymbol{C} $$
The area element inside the triangle is
$$ {\rm d}\,{\rm area}(i) = \frac{\partial {\rm pos}}{\partial u} \times \frac{\partial {\rm pos}}{\partial v} \; {\rm d}u \,{\rm d}v =
u \left( \boldsymbol{A} \times \boldsymbol{B} + \boldsymbol{B} \times \boldsymbol{C} + \boldsymbol{C} \times \boldsymbol{A} \right) {\rm d}u \,{\rm d}v $$ the integral of which results in (1)
$${\rm area} = \int \int u \left( \boldsymbol{A} \times \boldsymbol{B} + \boldsymbol{B} \times \boldsymbol{C} + \boldsymbol{C} \times \boldsymbol{A} \right) {\rm d}u \,{\rm d}v $$
It helps to reverse the above and substitute below $\left( \boldsymbol{A} \times \boldsymbol{B} + \boldsymbol{B} \times \boldsymbol{C} + \boldsymbol{C} \times \boldsymbol{A} \right) = 2\; {\rm area}$
The centroid the triangle is the average position
$$ {\rm cen} = \frac{ \int \int {\rm pos}(u,v) u \left( 2 {\rm area} \right) {\rm d}u \,{\rm d}v }{\rm area} = \frac{\boldsymbol{A}+\boldsymbol{B}+\boldsymbol{C}}{3} $$
Finally, the MMOI value is found by the integral
$${\rm mmoi} = \int \int \left({\rm pos}\cdot{\rm pos}\right) u \left( 2 {\rm area} \right) {\rm d}u \,{\rm d}v $$
which with some manipulation yields (3). Note that $$\small ({\rm pos}\cdot {\rm pos}) = (1-u)^2 (\boldsymbol{A}\cdot \boldsymbol{A}) + 2 u (1-u)(1-v) (\boldsymbol{A}\cdot \boldsymbol{B}) + u^2 (1-v)^2 (\boldsymbol{B}\cdot \boldsymbol{B}) + 2 u^2 v (1-v) (\boldsymbol{B}\cdot \boldsymbol{C}) + 2 u v (1-u) (\boldsymbol{C}\cdot \boldsymbol{A}) + u^2 v^2 (\boldsymbol{C}\cdot \boldsymbol{C})$$