The heart of this question stems from a desire for a mathematical "algebraic" definition of a dimensionful quantity.
The OP is concerned that a mathematical vector space $V$ should admit multiplication by elements of the mathematical field, $F$, over which the vector space is defined.
The problem is that dimensionful quantities do NOT form a field. Suppose $x$ and $y$ represent two lengths, that is they both have dimension of $L$. We might write
$$
[x] = [y] = [L]
$$
To notate this.
We are able to add $x$ and $y$ together. If we allow negative lengths then we see that there is an additive inverse for every length and there is also an additive identity length (zero length). The addition is commutative and associative and the operation is closed. This means that the space of physical lengths forms an abelian group.
However, we cannot multiply two lengths together and get another length. This is because
$$
[xy] = [L^2]
$$
If we multiply two lengths together we have something which has dimensions of length squared. This is like an area. Of course in physics we know that this is a perfectly valid and simple operation. We use it all the time. Mathematically, however, this means that the space of physical lengths is NOT a field, because for something to be a field you must be able to multiply two of its elements and get another one of its elements.
The OPs question then, if I may rephrase it, is how do we multiply a vector by a dimensionful quantity if dimensionful quantities don't form a field?
The physics answer is, it's easy, just multiply and add as you would if the dimensionful quantity were an element of a bonafide field and just keep track of the dimensions as you normally would. The OP, however desires an answer based on algebraic mathematical definitions.
My answer will include the introduction of a new sort of mathematical object which I would like to refer to as a general physical quantity. Suppose that we have 3 base dimensions, mass ($M$), length ($L$) and time ($T$) and suppose all dimensionful quantities could be formed from these 3. In real physics, using the SI system, for example, you may need to include more base dimensions but this set of 3 will suffice for now.
Any physical quantity can be described by a real number $q$ and a 3-tuple $d \in \mathbb{R}^3$ which describes the dimension of that quantity. The 3-tuple carries the exponents of the respective dimension for the quantity. Any physical quantity can be expressed as a tuple $(q, d)$. For example a length $[L]$ may be written as:
$$
x = (1, (0, 1, 0))\\
$$
A time $[T]$ may be written as
$$
t = (3, (0, 0, 1))
$$
While a force $[ML^2 T^{-2}]$ would be written as
$$
f = (0, (1, 2, -2))
$$
We call the set of all tuples $(q, d)$ the set of physical quantities and denote this space by $P$.
We can define a partial addition on the space $P$. Suppose $p_1 = (q_1, d_2)$ and $p_2 = (q_2, d_2)$. We have
$$
p_1 + p_2 = (q_1 + q_2, d_1)
$$
if $d_1 = d_2$. If $d_1 \neq d_2$ then the operation is undefined. This means the space $P$ is NOT a group under addition. However, if we restrict to elements of a fixed $d$ then we do have an abelian group.
We can define multiplication of any two elements of $P$.
$$
p_1 * p_2 = (q_1q_2, d_1 + d_2)
$$
We now have a "rigorous" definition of a physical quantity, $p \in P$.
We can now define multiplication of a vector times a physical quantity. Let $P$ be the space of physical quantities and let $V$ be a vector space of the field $\mathbb{C}$. We define the space of dimensionful vectors $V_D$ as the set of tuples $(v, d)$ where $v \in V$ is a vector and $d\in\mathbb{R}^3$ is a dimension representation as before. This space has similar rules for multiplication and addition of its elements as $P$. We first define multiplication by a physical quantity. Suppose $v_1 = (v_a, d_a) \in V_D$ and $p_b = (q_b, d_b) \in P$. We define
$$
p_b * v_1 = (q_b, d_b) * (v_a, d_a) = (q_b v_a, d_a + d_b) \in V_D
$$
Note we can also define multiplication by a regular scalar $z \in \mathbb{C}$ by
\begin{align}
z * v_1 = z * (v_a, d_a) = (z v_a, d_a)
\end{align}
Alternatively, if we had allowed physical quantities $p\in P$ to be have complex values (instead of just reals) we could identify the "dimensionless scalars" with physical quantities whose dimension $d = \boldsymbol{0} = (0, 0, 0)$.
We can define addition of two elements $v_1 = (v_a, d_a)$ and $v_2 = (v_b, d_b)$ by
\begin{align}
v_1 + v_2 =& (v_a, d_a) + (v_b, d_b) = (v_a + v_b, d_a)
\end{align}
Where the operation is again only defined if $d_a = d_b$.
Consider the set
$$
V_{d_0} = \{\bar{v} = (v, d)|v\in V \text{ and } d = d_0 \in \mathbb{R}^3\} \subset V_D
$$
This is the subset of $V_D$ which has fixed dimension $d_0$. It can be seen that this space is a vector space because it is closed under addition and multiplication by dimensionless scalars and has all the appropriate properties.
Ok, now we have dimensionful vector spaces. I think we now need to define dimensionful operators on these spaces to finally answer the OPs question. A dimensionful operator $X_1$ is a tuple $(X_a, d)$ where $d$ is a dimension tuple as before and $X_a:V\rightarrow V$ is an operator on $V$. This operator takes in dimensionful vectors and outputs dimensionful vectors:
$$
X_1: V_D \rightarrow V_D
$$
defined by (supposing $v_1 = (v_b, d_b)$)
$$
X_1 v_1 = (X_a, d_a)*(v_b, d_b) = ( X_a v_B, d_a + d_b)
$$
This should answer the OPs question of how quantum operators can be dimensionful. This answer also may be useful for designing software protocols which handle dimensionful quantities.
The OP has another question which is, if it is so complicated to handle dimensionful quantities why have I not seen this presentation anywhere else? The answer is that it is not complicated at all to handle dimensionful quantities as I described above. Just algebraically manipulate the quantities as if they were elements of a single field (just making sure not to add the quantities unless they have the same dimension) and then perform dimensional analysis as you normally would. All of this discussion adds basically no value to a physicist trying to calculate something and as long as you pay attention to dimensional analysis throughout your calculation, you will not have any problems.
Said yet one more way, physical quantities behave exactly like elements of a field except for the single fact that you cannot add quantities that have different dimensions. This whole post is basically a very very long winded way to encode that statement in mathematical definitions. The main thrust of the OP is to point out that physical quantities are not a field.
edit:
Adding more to respond to the OPs specific concerns about Hermitian operators. In quantum mechanics lets consider the kets $|\psi\rangle$ to be dimensionless vectors. This is not always the case but it will suffice for now. How does a dimensionful operator act on these kets?
Well, recall that a dimensionful operator $\bar{X} = (X, d)$ is an operator (or function if you prefer) on the space $V_D$ of dimensionful vectors. However, consider the action of this operator on a dimensionless vector $\bar{v} = (v, \boldsymbol{0})$
\begin{align}
\bar{X} * \bar{v} =& (X, d) * (v, \boldsymbol{0}) = (Xv, d+\boldsymbol{0}) = (Xv, d) \in V_d
\end{align}
So we see that if we restrict $\bar{X}$ maps the vector space $V_0$ to the vector space $V_d$. And in particular the mapping is
$$
\bar{X} * (v, \boldsymbol{0}) = (1, d) * (Xv, \boldsymbol{0})
$$
So it is exactly as if you had a simple dimensionless vectors and operators, you just have to remember multiply by the unit dimensionful quantity $(1, d)$ at the end.
Dimensionful hermitian operators. Consider hermitian operator $H$ on $V$. Consider then $\bar{H} = (H, d)$. Suppose $v$ is an eigenvector of $H$ with real eigenvalue $\lambda$: $Hv = \lambda v$.
\begin{align}
\bar{H} * \bar{v} =& (H, d) * (v, \boldsymbol{0}) = (Hv, d) = (1, d) * (\lambda v, \boldsymbol{0})\\
=& (1, d) * \lambda \bar{v} = (\lambda, d) * \bar{v}
\end{align}
This notation makes it look very suggestive that $\bar{v}$ is an eigenvector of $\bar{H}$ with dimensionful eigenvalue $(\lambda d)$. Of course this statement is not technically correct because $\bar{H}$ is not a mapping from a vector space to itself but it is damn close to one. The only difference is a trivial multiplication by the unit dimensionful quantity $(1, d)$.
To address the OPs question: At the extremely high levels of pedantry we are entertaining here operators in quantum mechanics are NOT technically hermitian operators. They do not act from a space onto itself. Rather, they might map dimenionsless vectors in $V_{\boldsymbol{0}}$ to dimensionful vector in $V_d$. However, they look similar enough to regular hermitian operators that the difference doesn't affect anything and we can generally think of them as being hermitian operators.
Note it might be useful to define the adjoint of a dimensionful operator as
$$
\bar{X}^{\dagger} = (X, d)^{\dagger} = (X^{\dagger}, d)
$$
This will allow one to work with inner products of dimensionful vectors etc.