Regarding the $E$-field at the location of discrete point Most textbooks define the $E$-field at the location of $q_{1}$$\\$

as: $$\textbf{E}=-\frac{q_{2}}{r_{21}^2}\hat{\textbf{r}}$$
which is finite. However, the field is also defined as the force per unit charge that a test charge would feel if it were to be placed somewhere, with all other charges held fixed.
 
And so the field at the position of $dq$ is: 
$$E=\frac{q_{1}}{r_{13}^2}\hat{\textbf{r}}-\frac{q_{2}}{r_{23}^2}\hat{\textbf{r}}$$
where $r_{13}$ and $r_{23}$ are the distances from $dq$ to $q_{1}$ and $q_{2}$ respectively. Now, if we want to find the field at $q_{1}$ we will need to shrink $r_{13}$ toward zero, but this will make the field infinite. 
So the question is, in reality is the field extremely big at $q_{1}$ or is it finite and equals the first value above?
 A: It just depends on what you are talking about. In the first case you are considering the field at $q_1$ due to just the charge $q_2$. In the second case you are looking at the field at the location of $\text dq$ caused by both $q_1$ and $q_2$. So you are looking at two different things. 
Indeed, $\text dq$ will feel an "infinite force" as it is brought closer and closer to $q_1$. And in your first case $q_1$ will feel an "infinite force" as it is brought closer and closer to $q_2$.
A: When particles get close to each other their behaviors and forces between them are governed by quantum mechanics and quantum field theory.
the short answer is the force does not become infinite. 
when you try to bring two charges too close to each other, (let's say two electrons) the quantum effects become important and many unusual things happen.
For example it would be impossible to bring two stationary electrons close to each other because you would not exactly know where the electrons are! This is called Heisenberg's uncertainty principle. 
The quantum field theory comes to the play if you apply a very huge force and shoot the electrons toward each other hoping that they collide. in this case new particles can be created. 
