However, when the capacitor is connected to a battery source, the potential difference remains the same, even when a dielectric is inserted inside the capacitors. What is the reason for this?
The voltage (potential difference) across the capacitor does initially change. It temporarily drops. Given the relationship
$$C=\frac{Q}{V}$$
Insertion of the dielectric increases the capacitance. Since the charge cannot change instantaneously, the increase in capacitance must be accompanied by a decrease in the capacitor voltage to below the battery emf (open circuit voltage). This causes the battery to charge the capacitor until its voltage equals the battery emf.
Keep in mind that all real batteries have internal resistance. So the battery charging current results in a voltage drop across the battery internal resistance. This means that during charging the battery terminal voltage is below the battery emf and matches the capacitor voltage.
Hope this helps.