I think you could not arrive at your desired Feynman rules because your second equation is wrong. You see, given your definition of the graviton field as a small perturbation to the Minkowski metric, we can rewrite
\begin{align*}
g^{\mu\nu} g^{\rho\sigma} F_{\mu\rho} F_{\nu\sigma} &\approx (\eta^{\mu\nu} \eta^{\rho\sigma} - \kappa h^{\mu\nu} \eta^{\rho\sigma} - \kappa h^{\rho\sigma} \eta^{\mu\nu}) F_{\mu\rho} F_{\nu\sigma} \\
&= F^2 + 2 \kappa {h^\mu}_\nu\ \partial_{[\mu}A_{\rho]} \partial^{[\rho}A^{\nu]}\,. \\
\end{align*}
Moreover, you made a mistake in your expansion of the invariant measure.
$$ \sqrt{-g} \approx 1 + \frac{\kappa}{2} \eta_{\alpha \beta}h^{\alpha \beta}$$
These considerations give your Interaction Lagrangian the following form:
$$ \boxed{\mathcal L_I = -\frac{\kappa}{4} \eta_{\alpha \beta}h^{\alpha \beta} \partial_{\mu}A_{\nu} \partial^{[\mu}A^{\nu]} - \frac{\kappa}2 {h^\mu}_\nu\ \partial_{[\mu}A_{\pi]} \partial^{[\pi}A^{\nu]}}\,. \tag4$$
You do not need to write your Lagrangian in the quadratic form (e.g. $A_\sigma \hat{N}^{\sigma\lambda}_{\mu\nu} A_\lambda$) unless, of course, you want to find the free propagator of your theory (which is $\hat{N}^{-1}$, if you have that free term in your Lagrangian, which you do not).
Note that there are six terms in the above Lagrangian which I will list down below.
- $-\frac{\kappa}{4} \eta_{\alpha \beta}\ h^{\alpha \beta}\ \partial_{\mu}A_{\nu}\ \partial^{\mu}A^{\nu}$
- $+\frac{\kappa}{4} \eta_{\alpha \beta}\ h^{\alpha \beta}\ \partial_{\mu}A_{\nu}\ \partial^{\nu}A^{\mu}$
- $- \frac{\kappa}2 h^{\mu\nu}\ \partial_{\mu}A_{\pi}\ \partial^{\pi}A_{\nu}$
- $+ \frac{\kappa}2 h^{\mu\nu}\ \partial_{\pi}A_{\mu}\ \partial^{\pi}A_{\nu}$
- $+ \frac{\kappa}2 h^{\mu\nu}\ \partial_{\mu}A_{\pi}\ \partial_{\nu}A^{\pi}$
- $- \frac{\kappa}2 h^{\mu\nu}\ \partial_{\pi}A_{\mu}\ \partial_{\nu}A^{\pi}$
For each of these terms, you can try to write down the momentum-space contribution to the vertex interaction,
as follows. Consider, without loss of generality, the first term from the above list: $-\frac{\kappa}{4} \eta_{\alpha \beta}\ h^{\alpha \beta}\ \partial_{\mu}A_{\nu}\ \partial^{\mu}A^{\nu}$.
- Step 1: Start with the constant coefficients of the interaction term (such as $-\frac{\kappa}{4} \eta_{\alpha \beta}$). Write them down with an extra factor of $i$, the imaginary number.
- Step 2: Assign to each field in the interaction term (such as $h^{\alpha \beta}$) a corresponding schematic in the Feynman diagram (such as $h^{\rho \sigma}$) and write down the Lorentz indices of the pairing in terms of Minkowski matrices (such as $\eta^{\alpha \rho}\eta^{\beta \sigma}$).
- Step 3: Write down the derivatives that act on a certain field (such as $\partial_{\mu}A_{\nu}$) as the momentum of the corresponding field in the Feynman diagram (such as $\eta_{\nu\alpha}{k_1}_\mu$ for choice of $A_\alpha$ as the representative schematic).
- Step 4: Add all possible contributions due to different choices of schematic assignments (such as $\eta_{\nu\beta}{k_2}_\mu$ for choice of $A_\beta$ as the representative schematic for $\partial_\mu A_\nu$).
The result should look like this.
$$ -\frac{i\kappa}{4} \eta_{\alpha \beta} \eta^{\alpha \rho}\eta^{\beta \sigma} (\eta_{\nu\alpha}{k_1}_\mu {\eta^\nu}_\beta {k_2}^\mu + \eta_{\nu\beta}{k_2}_\mu {\eta^\nu}_\alpha {k_1}^\mu) = -\frac{i\kappa}{2} (k_1 \cdot k_2) \ \eta^{\rho \sigma} \eta_{\alpha \beta} \tag{5.1}$$
You can consider the above steps as Feynman meta-rules to find the Feynman rules for any "well-behaved" theory. Try working out the other terms (#2 through #6) by yourself and see if you can get the following results from each of these terms.
$$ + \frac{i\kappa}{2} \eta^{\rho \sigma} {k_1}_\beta {k_2}_\alpha \tag{5.2}$$
$$ - \frac{i\kappa}{2} ({\eta_\alpha}^\sigma {k_2}^\rho {k_1}_\beta + {\eta_\beta}^\sigma {k_1}^\rho {k_2}_\alpha) \tag{5.3}$$
$$ + \frac{i\kappa}{2} (k_1 \cdot k_2)\ {\eta^\rho}_{(\alpha} {\eta_{\beta)}}^\sigma \tag{5.4}$$
$$ + \frac{i\kappa}{2} \eta_{\alpha\beta} {k_1}^{(\rho} {k_2}^{\sigma)} \tag{5.5}$$
$$ - \frac{i\kappa}{2} ({\eta_\alpha}^\rho {k_2}^\sigma {k_1}_\beta + {\eta_\beta}^\rho {k_1}^\sigma {k_2}_\alpha) \tag{5.6}$$
These are the ten terms that you wanted to have (eq. 3). I hope that helps. Please comment below if you do not understand something and need further clarification.
NOTE:
To understand why the meta-rules work the way they do, please read a standard book on Quantum Field Theory. I would recommend A. Zee: Quantum Field Theory in a Nutshell, Ch. 1.7. You will also find this resource quite useful.