MY thoughts: I think it will not be as the expression for RMS value is dependent only on the peak value of current. Is my thinking correct?
The capacitance will increase when the dielectric is inserted because the relative permittivity of the material compared to air is likely to be higher. However, the current could reduce (or increase) depending how you have the LCR circuit configured.
If you are applying a frequency that corresponds to the resonant frequency of the LCR then maximum current will be circulating in the capacitor and inductor (parallel tuning). If you increased the capacitance this would mean a decrease in current because the LCR is now non-ideally "tuned" to the input frequency.
The reverse could happen if F corresponds to the new resonant frequency after dielectric material is inserted into the capacitor.
You missed something, When you insert a dielectric into capacitor it is capacitance increases. When the capacitance increases, what happens to the peak value of the current? Think about that
Will the rms current be affected by inserting dielectric in the capacitor in an LCR circuit?
The answer is: maybe. You haven't given enough context to give a more definitive answer than that.
What one can say for sure is that inserting a dielectric between the plates of an air capacitor will increase the capacitance (assuming no other changes).
This means that the AC current through, for a given AC voltage across, the capacitor will increase.
But, consider a series LCR circuit driven by an AC current source. Clearly, the current through the capacitor is independent of the capacitance (the current source fixes the current through the capacitor). In this case, the AC voltage across the capacitor will decrease.
So, without more context or a circuit diagram, it's not possible to give a yes or no answer to your question.