Don't the four quantum numbers make two electrons distinguishable? From the Pauli's Exclusion Principle no two electrons in a bound system have all same quantum numbers. This means that an electron can be uniquely specified by the four quantum numbers and hence can be distinguished from others.
I understand we still will not be be able to tell 'which electron' has that set of quantum numbers. So for instance, we cannot tell apart ; electron $1$ having $\{n,m,l,+s\}$ and electron $2$ having $\{n,m,l,-s\}$ from electron $1$ having $\{n,m,l,-s\}$ and electron $2$ having $\{n,m,l,+s\}$.
But why does that matter? We don't need 'electron-$1$' and 'electron-$2$' labels . We can just tag the electron as 'electron -$\{n,m,l,+s\}$' and so on.
To J Murray's comment:
This is from the Tony Genault book on statistical mechanics and this is what I meant in my comment.

In this chapter we shall treat the other type of assembly, in which the particles
are distinguishable. The physical example is that of a solid rather than that of a gas.
Consider a simple solid which is made up of N identical atoms. It remains true that the
atoms themselves are indistinguishable. However, a good description of our assembly
is to think about the solid as a set of N lattice sites, in which each lattice site contains
an atom. A ‘particle’ of the assembly then becomes ‘the atom at lattice site 4357
(or whatever)’. (Which of the atoms is at this site is not specified.) The particle is
distinguished not by the identity of the atom, but by the distinct location of each
lattice site. A solid is an assembly of localized particles, and it is this locality which
makes the particles distinguishable.

I am interested specifically in the line:

The particle is
distinguished not by the identity of the atom, but by the distinct location of each

I can simply rephrase it as:
"The particle is
distinguished not by the identity of the atom, but by the distinct quantum numbers of each."
Since even in the initial case of solid, we don't care whether its the same atom at a particular lattice when we look at it the second time but that it is the 'atom at lattice 1435 etc'
 A: 
So, from the Pauli's Exclusion Principle no two electrons in a bound system have all same quantum numbers. This means that an electron can be uniquely specified by the four quantum numbers and hence can be distinguished from others. I understand we still will not be be able to tell 'which electron' has that set of quantum numbers.

Your last sentence is what we mean when we say that two particles are indistinguishable.  More precisely, two particles are called indistinguishable if swapping their quantum numbers doesn't affect the state of the composite system, or equivalently if swapping their quantum numbers only causes an overall phase shift of the composite wavefunction, $\Psi \mapsto e^{i\theta}\Psi$ for some $\theta$.
The spin-statistics theorem says that under broad conditions, there are only two types of particle - those for which $\theta =0$, which we call bosons, and those for which $\theta = \pi$, which we call fermions.  Under certain conditions (e.g. in 2D) this does not hold, but we'll put that aside for now.
The point is that indistinguishability doesn't mean that every particle in a system has the same properties, but rather that if any two particles are interchanged, the state of the composite system is unaffected.  In the specific case of fermions, this means that $\Psi \mapsto-\Psi$ when two fermions are interchanged; if they were to reside in the same state, we would also have that $\Psi\mapsto \Psi$ (since nothing has changed), which implies that $\Psi=0$.  As a result, it follows that a system of indistinguishable fermions cannot have any two particles in the same state - i.e. the Pauli exclusion principle.
A: The distinguishability requires not only associating partciles with a different sets of quantum numbers, but also tracing these particles. In classical physics we can follow the trajectories of the two particles and knwo which came from which initial position. However, the two electrons in the OP could switch their places and we would never know this.
However, note that there is a more serious flaw with the reasoning in the OP: it assumes the validity of the Pauli principle, which follows from the idnistinguishability of the electrons, to question this very indistinguishability. In other words, this is a circular reasoning.
