We can't remove the resistor between the two points we've chosen because they're not at the same voltage.
OK, let's unpack that a little. Imagine that you actually have a resistor network (any resistor network) built and want to measure its resistance with an ohmmeter. To do that, you need to choose two of the points in the network and connect the leads of the ohmmeter to them. The ohmmeter will then pass a small DC current through the network, measure the voltage difference $\Delta V$ between its leads and the current $I$ flowing through the network between them, and calculate the resistance $R$ of the network using Ohm's law: $$R = \frac{\Delta V}I$$
Now, since we're feeding a fixed DC current into the network, and since we only have passive resistive components in it, the network will very rapidly (essentially instantaneously) settle into a steady state where each node is at a constant voltage and each resistor has a constant current flowing through it.
Specifically, the node to which we've connected the negative lead of the ohmmeter is pulled down to some fixed voltage $V^-$, while the node to which we've connected the positive lead is pulled up to some voltage $V^+ > V^-$. Every other node $i$ of the network will be at some intermediate voltage $V_i$ between $V^-$ and $V^+$. Using your colored balls analogy, it's as if we've chosen two balls, colored one of them white and one black, and colored the rest of the balls with different shades of gray according to their equilibrium voltage, as determined by Ohm's law and Kirchhoff's first law.
Indeed, we can mechanically solve for the equilibrium current $I$ through the system simply by writing down the expressions for the current flow through each resistor $ij$ given by Ohm's law: $$I_{ij} = \frac{V_i - V_j}{R_{ij}}$$ and the conservation of current at each node $i$ given by Kirchhoff's first law: $$\sum_j I_{ij} = I^0_i = 0 \text{ for all }i \notin \{a,b\},$$ together with the known voltages $V_a = V^+$ and $V_b = V^-$ and excess currents $I^0_a = -I^0_b = I$ for the endpoint nodes $a$ and $b$, and solving the resulting system of linear equations for $I$.
However, if we want to simplify the system before solving it, we can apply two useful observations:
First, if two nodes have the same voltage, no current can flow between them: $V_i = V_j$ $\implies$ $I_{ij} = 0$. (Check this using Ohm's law above!) Thus, we can completely ignore any resistors between such nodes. In fact, we can even effectively collapse such nodes together into a single node (as if they were connected by a wire with zero resistance), as long as we remember to account for the fact that we may end up with several resistors in parallel between two nodes.
Second, if we have two nodes $i$ and $j$ such that $R_{ik} = R_{jk}$ for all nodes $k$ (where we take $R_{ik} = \infty$ if $i$ and $k$ are not connected) and $I^0_i = I^0_j$, then we can swap the labels of those two nodes without changing any of the parameters of the system. Thus, by symmetry, the solution must have $V_i = V_j$, since otherwise swapping the labels would change the solution without changing the parameters (which is a contradiction if the system is well defined and thus uniquely solvable).
In your example network, every node is connected to every other node by identical resistors, and so $R_{ik} = R_{jk}$ for all nodes $i$, $j$, $k$. For all but the two chosen endpoint nodes, we also have $I^0_i = I^0_j = 0$, and so all the other nodes except the endpoints can be exchanged without changing the system. Thus, we may ignore any resistors between them and even collapse them all into a single node.
However, the reason we cannot exchange the two chosen endpoints is that we broke the symmetry when we connected the measurement leads to them: those points have current flowing into them from outside the network, which will pull them to different voltages. In particular, the voltage difference will cause a non-zero current to flow across any resistor connecting those two nodes, and so such resistors cannot be ignored when calculating the total current flow across the system.
(If there was no externally supplied current, all the nodes in the network would indeed be symmetrical, and we could correctly deduce that no current would flow between any of them. But that scenario is completely useless for calculating the resistance, since we'd just end up with the indeterminate form $R = \Delta V/I = 0/0$.)