Edit: I found a text that mentions that when the incident medium is absorbing, the usual equations for interference don't apply, so I have refined my question in the title.
I have been working on creating a Blender simulation for thin-film interference using the complex versions of the Fresnel and interference equations.
Part of the implementation of the simulation includes reversing the order of the layers when we see backfacing geometry, such that the substrate becomes the medium and the medium becomes the substrate in the calculations. The reasoning is that light also exits through the (transparent) material. That's where the problem comes in.
These reversed layers do calculate correctly for non-absorbing substrates, but the moment a non-zero extinction coefficient is entered, it starts spitting out values greater than 1 for both reflectance and transmittance.
In the first implementation of the equations, I used
$$ \hat a_x = \hat n_x cos(\theta_x) = \sqrt{ \hat n_x^2 - \hat n_1^2 sin(\theta_1) } \ , \; \hat n = n + ik$$
within the Fresnel equations
$$ \begin{align} \hat r_{s|ij} & = \frac{ \hat a_i - \hat a_j }{ \hat a_i + \hat a_j } \\ \hat r_{p|ij} & = \frac{ \hat n_j^2 \ \hat a_i - \hat n_i^2 \ \hat a_j }{ \hat n_j^2 \ \hat a_i + \hat n_i^2 \ \hat a_j } \\ \hat t_{s|ij} & = \frac{ 2 \hat a_i }{ \hat a_i + \hat a_j } \\ \hat t_{p|ij} & = \frac{ 2 \hat n_i \hat n_j \hat a_i }{ \hat n_j^2 \ \hat a_i + \hat n_i^2 \ \hat a_j } \\ \end{align} $$
and within the interference equations
$$ \begin{align} \hat r & = \frac{\hat r_{12} + \hat r_{23} e^{i 2 \beta}}{1 + \hat r_{12} \hat r_{23} e^{i 2 \beta}} \\ \hat t & = \frac{\hat t_{12} \hat t_{23} e^{i \beta}}{1 + \hat r_{12} \hat r_{23} e^{i 2 \beta}} \\ \beta & = \frac {2 \pi t}{\lambda} \hat a_{film} \end{align} $$
Taking the calculations to Excel, I found that something about these equations were causing values for reflectance and transmittance to be greater than 1 (I even saw values of 14 for reflectance!). These values generally increased with the thickness.
After more investigation, I found that when the thickness is zero (thereby eliminating the film), the flipped layers do calculate correctly. This lead me to believe that the OPD $(\beta)$ used in the interference calculations is the problem.
Looking around for possible solutions, for which there seemed to be no content, I found this spectral reflectance calculator which showed that there was a way to calculate it correctly. There's no way to actually enter complex refractive indices into the calculator, so I simply used the provided metals to test the flipped layer idea. I also used it to corroborate the results of my implementation.
Wanting to test another method of interference calculations, I also created an implementation that uses the 2x2 matrix method, but the results were the same.
More searching brought me to Born and Wolf's Principles of Optics (7th Edition) where they derive calculations for absorbing films on a transparent surface. Using their derivation of the Fresnel and interference equations for that specific situation as a guide for my purposes, the following was found:
$$ \hat n_x cos(\theta_x) = u_x + iv_x $$
where
$$ \begin{align} 2u_x^2 & = a_x + \sqrt{ a_x^2 + 4 b_x^2 } \\ 2v_x^2 & = -a_x + \sqrt{ a_x^2 + 4 b_x^2 } \end{align} $$
and
$$ \begin{align} a_x & = n_x^2 - k_x^2 - (n_1^2 - k_1^2) sin(\theta_1) \\ b_x & = n_x k_x - n_1 k_1 sin(\theta_1) \end{align} $$
Using $ u_x + iv_x $ instead of $ \hat a_x $ in the Fresnel equations and the OPD, the reflectances and transmittances do not reach as high as before. In fact, I have yet to find values that reach above 1. Although, when checking for energy conservation by adding the values together $ (R + T) $, the result is greater than 1 at and close to normal incidence.
Comparing the usage of $ u_x + iv_x $ to $ \hat a_x $ showed that the problem lies in how the imaginary part is calculated; switching back and forth between the real parts showed no change.
To confirm whether the OPD was the problem, I used the original Fresnel equations with the new OPD equation, and I also used the old OPD with the new Fresnel equations. This showed exactly the same results as what resulted from the new equations and old equations respectively, suggesting the problem does lie in the OPD. (Edit: it seems that the OPD may not truly be the problem as described by the note I added at the top of this post)
Also, the reflectance calculator seems to suggest that, unless the film is also absorbing, there is no absorbance when light passes out of an absorbing substrate into the medium. I also found another calculator, but it gave exactly the same results as my new calculations.
How do I go about performing interference calculations for thin-films with transparent, slightly absorbing substrates when light is exiting the material?
The following are some of the results from the simulation.
Results of the equations using $ \hat a $: (Left: $R+T$ Color of the material, Right: Full material)
$$ \hat n_{sub} = 1.5 + i 0.1, \ \hat n_{film} = 2 + 0i, \ t = 500nm $$
Results of the equations using $ u + i v $: (Left: $R+T$ Color of the material, Right: Full material)
$$ \hat n_{sub} = 1.5 + i 0.1, \ \hat n_{film} = 2 + 0i, \ t = 500nm $$
The following show whether $R+T$ is greater than one. If true, it returns 1 for whichever of the R, G, B values is greater than 1 to make a color. Here's a link to animations of these values changing as thickness increases. (Left: Equations using $ \hat a $, Right: Equations using $ u + i v $)
Edit: I realized that $\hat a$ can give the same results as using $u + iv$ because $u + iv$ is simply the other solution to the square root.
I also was able to implement another version of the interference equation where each Fresnel coefficient has been converted to it's exponential form:
$$ \hat r_{ij} = |\hat r_{ij}| e^{i(arg(\hat r_{ij}))} = \rho_{ij} e^{i \phi_{ij}}$$ $$ \hat t_{ij} = |\hat t_{ij}| e^{i(arg(\hat t_{ij}))} = \tau_{ij} e^{i \chi_{ij}}$$
such that the new equations become:
$$ \begin{align} \hat r & = \frac{\rho_{12} e^{i \phi_{12}} + \rho_{23} e^{i (2 \beta + \phi_{23})}}{1 + \rho_{12} \rho_{23} e^{i (2 \beta + \phi_{12} + \phi_{23})}} \\ \hat t & = \frac{\tau_{12} \tau_{23} e^{i (\beta + \chi_{12} + \chi_{23})}}{1 + \rho_{12} \rho_{23} e^{i (2 \beta + \phi_{12} + \phi_{23})}} \end{align} $$
These equations unfortunately give the same results as in the second pair of images above. I guess it's to be expected when it's simply just another form of the same thing.