Why aren't resistors equivalent to breaks when $R\to\infty$? 
Find the voltage across each resistor as $R\to\infty$


Kirchoff's voltage law gives
$$10\ \mathrm{V} - V_R-V_R = 0 \implies V_R = 5\ \mathrm{V}$$
However, don't we get two holes in the circuit as the resistances approach infinity ? The dangling wire in the middle is confusing me a lot.

 A: I think the question is quite interesting actually. 
When resistances are equal the voltage will divide itself equally because the resistances are coupled in series. Then we take each R to infinity and we assume that we do this to each resistance in the same manner. I think that the voltage across each resistance remains 5 Volts in this case when you take R to infinity. 
I think your second drawing is good and shows that you have a good intuition what is happening. Essentially when you let the R go to infinity you get something like a capacitor! This is essentially what you have drawn. It then also becomes clear I hope that indeed there should be 5 Volts across each 'capacitor'.
A: I would try a more "dynamic" approach, where you actually have an RC circuit (https://en.wikipedia.org/wiki/RC_circuit ) - even a piece of wire has some non-zero capacitance. The typical time scale of transient processes in such circuits is $RC$. When $R\rightarrow \infty$, you have $RC\rightarrow \infty$, so you have to wait longer and longer for transients to subside. If, however, $R=\infty$, the transients will never subside, so if there was an initial charge (and, therefore, voltage) between the resistors, it will never change.
A: If you send $R$ to infinity that's equivalent to an open circuit. Since an open circuit, as you can tell from the name, is not a closed loop, Kirchoff's law is not valid, hence your "paradox".
Edit: Kirchoff's law is valid, but not in the form you wrote it. You cannot say that $V_R=5 V$ because when you say that $R \to \infty$ what you are saying is that it is an open circuit.
A: In the context of ideal circuit theory, the voltage across each resistor is given by voltage division:
$$V_{R1} = 10V \cdot \frac{R_1}{R_1 + R_2}$$
$$V_{R2} = 10V \cdot \frac{R_2}{R_1 + R_2}$$
If we set $R_2 = \alpha R_1$, these equations become
$$V_{R1} = 10V \cdot \frac{1}{1 + \alpha}$$
$$V_{R2} = 10V \cdot \frac{\alpha}{1 + \alpha}$$
which is valid for $R_1 > 0$ and in the limit as $R_1 \rightarrow \infty$ and this is really all there is to it.
However, your second drawing isn't equivalent since it does not contain as much information.  We see two ideal open circuits in series which means that the voltage across each is indeterminate; we only require that the sum of the voltage across each equals $10\mathrm{V}$.
But this result is essentially academic this isn't an adequate model of a physical circuit.  In reality, there are inescapable inter-conductor capacitance that are not modelled here.
In addition, physical resistors have internal ('parasitic') capacitance and inductance (the circuit elements in the equivalent circuit below are understood to be ideal)

For typical resistor values and low enough frequencies, the internal capacitances are insignificant.
So in your circuit thought experiment, let's try this equivalent circuit model for the resistors and see that, as $R$ is increased without bound, we are left with capacitors between the leads, not genuine open circuits; there is a voltage across each capacitor which remains even when $R$ is taken to infinity (this isn't intended to be rigorous but rather to give you some idea of how to think about problems like this).
However, note that if one were to try to measure the voltage across either capacitance with, e.g., an ordinary multimeter, the (non-infinite) input impedance of the multimeter would be placed in parallel with the capacitance which drastically changes the circuit under test.
A: As R reaches infinity you will find new voltages appearing between R1 and R2, the result of induced interference from radio broadcast and other like environmental noise(s).  As that soon to be free floating wire is more and more isolated it's less held back by the rest of the circuit and starts to behave like the free wire section it will soon become.  Expect the section to begin to act as a radio antenna.
