What makes a system of units self-consistent? What are the rules governing the creation of a self-consistent system of units? To be clear, I'm not asking about making units universal or replicable, I'm asking about the mathematics governing what makes a valid system of units valid.
 A: Your question may be answered within this mathematical note on Buckingham's pi-theorem. It describes the mathematics and consequences of a consistent system of units. The author of this reference provides the following insights that may be relevant to your question :
Suppose you have a set $\{ F_i \}$ of fundamental units, where $i=1\ldots m$. This could, for example, be the seven fundamental SI-units. Then any physical quantity $R$ can be expressed as
$$ R = \rho\cdot [R],\quad (1)$$
where $\rho$ is a number and
$$ [R] = \prod_{i=1}^m F_i^{a_i}. \quad (2)$$
are the units of $R$.
The main formal requirement for the system of units is independence. Independence of the fundamental units can be mathematically stated as
$$ \mbox{If } \prod_{i=1}^m F_i^{x_i} = 1,\quad\mbox{it follows that }x_1=\ldots=x_m=0.\quad (3)$$
I would argue that a system of units is valid (and self-consistent), if it fulfills the independence criterion (3), and allows us to express all physical quantities relevant to us in the form (1) with units given by (2).
Note also, that Buckingham's pi-theorem actually tells us that we can do theoretical physics without units using only dimensionless quantities and relations between them. It also shows us that experimentalists can always present their data in a meaningful way using dimensionless parameters, i.e., experimental parameters scaled to the natural scales of the physics problem.
A related post on Physics SE is found here.
