Coulomb force between two point charges in a dielectric medium In a dielectric medium with relative dielectric constant $\epsilon_r$, what is the Coulomb force between two free point charges $q_1$ and $q_2$ at distance $r$? Is it equal to the Coulomb force in vacuum divided by $\epsilon_r$ or $\epsilon_r^2$, i.e., whether the formula is
$$F=\frac{q_1q_2}{4\pi\epsilon_0\epsilon_rr^2}\quad\mbox{or}\quad\frac{q_1q_2}{4\pi\epsilon_0\epsilon_r^2r^2}?$$
I know in a dielectric medium, we have $\nabla^2\phi=-\rho_0/(\epsilon_0\epsilon_r)$. Assuming the medium is infinitely big with no boundary to consider, the Coulomb field generated by either $q_1$ or $q_2$ is reduced by a factor of $\epsilon_r$. But I also know this effect is due to the bound charges $-q_1(1-1/\epsilon_r)$ and $-q_2(1-1/\epsilon_r)$ that surround the free charges $q_1$ and $q_2$, leaving net charges $q_1/\epsilon_r$ and $q_2/\epsilon_r$. So let's say if the $q_1$ and $q_2$ are like charges and I connect them with an insulating rope. Which force formula is correct to use, if I want to calculate the tension in the rope at equilibrium, assuming the medium is a frictionless fluid?
 A: The Coulomb force in a medium with relative dielectric constant $\epsilon_r$ is given by your first equation. Only from this follows the electric field strength of a spherical symmetric free charge $Q$ in the dielectric with $$E=\frac{Q}{4\pi\epsilon_0\epsilon_r r^2} \tag{1}$$ which, with the electric displacement $D=\epsilon_r \epsilon_0 E$,  results in the correct Gauss Law $$ \int_{sphere} \epsilon_r \epsilon_0 E da=Q \tag{2}$$ This is equivalent to the differential form of Gauss's Law, the Maxwell equation in a dielectric $$ div (\epsilon_r \epsilon_0 \vec E)=\rho$$ where $\rho$ is the free charge density.
Note added after a comment by Zhouran He: In Coulomb's Law for the electric force $F$ exerted by a free charge $q_1$ on a second (test) charge $q_2$ in a dielectric with relative permittivity $\epsilon_r$, only the charge $q_1$ as the source of the force field can be considered to be reduced by the polarization charges of the dielectric to the $q_1/\epsilon_r$ so that the vacuum Coulomb law can be used with  this net charge. Even though the charge $q_2$ is also surrounded by polarization charges, the force $F$ exerted by the net charge $q_1/\epsilon_r$ works on the free charge $q_2$. One can alternatively consider $q_2/\epsilon_r$ to be the net charge exerting the force $F$ on the free (test) charge $q_1$. The free charge $q_2$ sees a net charge $q_1/\epsilon_r$ exerting a force $F$ on it according to Coulombs vacuum law. The polarization charges induced by itself around it don't exert a force on itself. The same reasoning applies with interchanged roles of the charges. Thus the second form of Coulombs Law for a dielectric is correct.  
A: The force analysis of the problem is done thanks to @freecharly. I now work it out using two other methods: virtual work and field energy. Suppose charge $q_1$ is fixed and charge $q_2$ moves along the rope by a small virtual displacement $\delta r$ away from $q_1$. The Coulomb field due to the net charge $q_1/\epsilon_r$ repels $q_2$ and attracts the bound charge $-q_2(1-1/\epsilon_r)$ surrounding $q_2$. When $q_2$ moves by the distance $\delta r$, the bound charge doesn't really move with $q_2$. By definition, bound charge cannot move. What actually happens is that the bound charge at the original position of $q_2$ depolarizes to neutrality, while some new bound charge of the same amount reappears at the new position of $q_2$. So no work is done to the bound charge $-q_2(1-1/\epsilon_r)$ because no bound charge actually moved the distance $\delta r$. Therefore,
$$F\delta r=\frac{q_1q_2}{4\pi\epsilon_0\epsilon_rr^2}\delta r,$$
which means the first formula is correct. Notice that the visionary "displacement" of the bound charges $-q_2(1-1/\epsilon_r)$ is not a real displacement. Therefore no work is done to them.

The field energy method does not distinguish free charges and bound charges or track how charges move. It uses the energy of capacitors $\,W=\frac{1}{2}CU^2\,$ with $\,C=\epsilon_0\epsilon_rS/d\,$ and $\,E=U/d\,$ to obtain
$$W=\frac{1}{2}\epsilon_0\epsilon_rE^2Sd.$$
Therefore the energy density of an $E$-field in a dielectric medium is greater than the same $E$-field in vacuum by a factor of $\epsilon_r$ due to the polarization of the medium. When the distance between the net charges $q_1/\epsilon_r$ and $q_2/\epsilon_r$ increases by $\delta r$, if these were charges in vacuum, the $E$-field energy would reduce by an amount equal to the work done by the force in the second formula times $\delta r$. But now the $E$-field of the net charges $q_1/\epsilon_r$ and $q_2/\epsilon_r$ are in the medium. Therefore, the work done is $\epsilon_r$ times greater. In formula we have
$$F\delta r=\frac{(q_1/\epsilon_r)(q_2/\epsilon_r)}{4\pi\epsilon_0r^2}\delta r\times\epsilon_r=\frac{q_1q_2}{4\pi\epsilon_0\epsilon_rr^2}\delta r.$$
$E$-fields in dielectric media contain more energy than the same $E$-fields in vacuum by $\epsilon_r$ times.
