OK, I figured out all my mistakes and the results seem sensible now.
I was using wrong formulas and, first of all, was not using imperial units which is necessary with UIUC's data. The right formulas are:
$F_T(RPS) = C_T \rho n^2 D^4$,
where: $\rho [\frac{slugs}{ft^3}] = 0,00238$ -$\rho [\frac{slugs}{ft^3}] = 0.0023769$ is air density, $n[RPS]$ -is prop angular speed in revolutions per seconds, $D[ft] = \frac{10}{12}$ -is prop diameter. For calculating torque, the below formula must be used:
$\tau(RPS) = \frac{C_P \rho n^2 D^5}{2 \pi}$.
The results obtained seem sensible and in synch with e.g. Drive Calculator data. Some graphs and screenshots below. Of course, the results are all in imperial units, I converted them to SI for graphs.