Consider a dilation of space $x\mapsto ax$ for some non-vanishing number $a$. Let $Q$ be the position operator defined by $(Q\psi)(x)=x\psi(x)$ on function $\psi$ of space. Suppose $\psi$ transforms like a scalar under these dilations, i.e. $\psi\mapsto\psi'$ at $\psi'(x)=\psi(x/a)$. Then how does $Q\psi$ transform? My intuition is that $Q\psi\mapsto Q\psi'$, where, to be explicit, $Q\psi'(x)=x\psi'(x)=x\psi(x/a)$. Is this correct? In a solution to a homework problem I require that is transforms into $ax\psi(x/a)$, which make no sense to me.
Suppose that instead of keeping track of one single function $x\psi(x)$ you were actually keeping track of a "tuple" (ordered pair) of functions $(f(x) ,g(x))$ where $f(x)=x$ and $g(x)=\psi(x)$.
You already have stated that $f(x)=x$ has to transform into $$ ax = af(x)\;, $$ so apparently $f(x)$ is not a scalar function. (It's kind transforming more like a vector-valued function might transform under a rotation)
You have also already stated that $g(x)=\psi(x)$ has to transform into $$ \psi(x/a)=g(x/a)\;, $$ so, $g$ is a scalar, since it is just identically equal to $\psi$, which is a scalar.
This means the tuple $(f(x),g(x))$ transforms into $$ (af(x),g(x/a)) $$
If I feel like it, I can always just take my tuple and multiply the two components together. Nothing stops me from doing this and the simple act of multiplication doesn't "know" about any transformation. So, I can define: $$ h(x) = MULTIPLY(f(x),g(x)) = f(x)*g(x) = x\psi(x)\;. $$ By the above-specified rules this transforms into $$ MULTIPLY(af(x),g(x/a)= af(x)*g(x/a) = ax\psi(x/a)\;. $$
Renaming your $(Q\psi)(x)$ to $h(x)$ shows explicitly by the above-argument that $(Q\psi)(x)=x\psi(x)$ transforms into $ax\psi(x/a)$.
As a mechanical rule, it looks like you can just remember that whenever there is an explicit factor of $x$ you can just replace it with $ax$ and whenever there is an explicitly scalar function $\psi(x)$ you can just replace it with $\psi(x/a)$ to effect the transformation. For example $x^2\psi(x) \to a^2x^2\psi(x/a)$