Rigorous mathematical definition of vector operator? In standard quantum mechanics textbooks, the concept of operators is often introduced as linear maps that map a Hilbert space $H$ onto itself:
$$
\hat{O}: H \rightarrow H \, .
$$
However, directly after, we use the position operator $\hat{\vec{x}}$, which isn't of the said shape, but instead is like a triple of operators, e.g. $\hat{\vec{x}} = (\hat{x}, \hat{y}, \hat{z})$. What I thought now is that maybe, one could treat the position operator as a linear map
$$
\hat{\vec{x}}:H \rightarrow H \otimes R^3 \, .
$$
Is it wrong to say the that the position operator is that kind of map? Does it make sense?
Edit: I know that you can treat the position operator as three independent operators $\hat{x}$, $\hat{y}$, $\hat{z}$. I just want to know if my way of treating the position operator is an equivalent way to see things, or if it is wrong to treat it like that. If it is wrong, why is it wrong?
 A: So there's a natural isomorphism
$$
\varphi: H^{\oplus 3} \to H \otimes \mathbb R^3\\
(a, b, c) \mapsto a \otimes e_0 + b \otimes e_1 + c \otimes e_2
$$
where $e_i$ is a basis for $\mathbb R^3$.
The definition you're objecting to is
$$
\hat{\vec{x}}: H \to H^{\oplus 3}\\
v \mapsto (\hat x(v), ~ \hat y(v) , ~\hat z(v))
$$
You can convert it to your version by composing with the above isomorphism. 
$$
\tilde x \equiv \varphi \circ \hat{\vec{x}}:~~ H \to H \otimes \mathbb R^3\\
v \mapsto \hat x(v) \otimes e_0 + ~ \hat y(v) \otimes e_1 + ~\hat z(v)\otimes e_2
$$
So your definition makes complete sense.
Comments
So this answers the question as stated. Judging by your comments on other answers I think you're looking for a nicer way to formalize the annoying statement 'blah operator transforms under blah group like a vector'. Whether this definition yields what you want should be the subject of another question.
Namely I think you're hoping somehow conjugation by a representation of the group will factor through your tensor product and have the group action on $\mathbb R^3$ just act on the second factor. Spent a couple of minutes trying to naively get this to work without much luck, will update if there's progress, or at least give a better explanation of why not.
Edit
I'm going to describe how this construct makes the "covariance under conjugation by some representation of some action on $\mathbb R^3$" condition more explicit, as OP had hoped.
For clarity let's start by giving a name to the operation of conjugating this "vector operator".
Let $G$ be some group that admits an action on $\mathbb R^3$. Let $U: G \to H$ be a unitary representation of $G$ on $H$. Define, for any $R \in G$
\begin{align}
C_R: \mathrm{Mor}(H, H^{\oplus 3}) &\to \mathrm{Mor}(H, H^{\oplus 3})\\
f(\cdot) &\mapsto (U(R) f(\cdot)^i U(R)^\dagger)_{i \in (0, 1, 2)}
\end{align}
where $\mathrm{Mor}(V, W)$ is collection of linear maps $V \to W$. This is just a name for  component-wise conjugation of this tuple of operators.
Now the main claim
Proposition
The statement
$$
C_R(\hat{\vec{x}}) = \left(\sum_{j}R_{ij}\hat{\vec{x}}(v)^j\right)_{i \in (0, 1, 2)}
$$
is equivalent to
$$
\varphi \circ  C_R(\hat{\vec{x}}) = (1 \otimes R)(\varphi \circ \hat{\vec{x}})
$$
Proof
For brevity we give the $\implies$ direction, the other direction should be obvious from the same computation
\begin{align}
\varphi\left(\left(\sum_{j}R_{ij}\hat{\vec{x}}(v)^j\right)_{i \in (0, 1, 2)}\right)
&= \sum_i\left(\sum_{j} R_{ij} \hat{\vec{x}}(v)^j\right) \otimes e_i \\
&= \sum_{ij}  \hat{\vec{x}}(v)^j \otimes R_{ij}e_i\\
&=\sum_{j}  \left(\hat{\vec{x}}(v)^j \otimes \sum_i R_{ij}e_i\right)\\
&= (1 \otimes R)\left(\sum_{j} \hat{\vec{x}}^j(v) \otimes e_j \right)\\
&=(1 \otimes R)\left(\varphi(\hat{\vec{x}(v)}) \right)
\end{align}
A: No. In three dimensions there are three position operators, $\hat{x}_1$, $\hat{x}_2$, and $\hat{x}_3$, or maybe $\hat{x}$, $\hat{y}$, and $\hat{z}$. Each of these is a linear operator in the first, correct sense. Each one maps states in the Hilbert space onto other maps in the Hilbert space and nothing more.
Now, the three distinct position operators are actually closely related to each other, so we often write $\hat{\vec{x}}$ as a shorthand for talking about all of them at once, but they are still three separate operators that map $H\rightarrow H$.
Also, when you have three related position operators the Hilbert space does carry additional structure compared to when you have just one position operator. Specifically, the Hilbert space becomes $L_2(\mathbb{R}^3,\mathbb{C})$, rather than just $L_2(\mathbb{R},\mathbb{C})$. This is a larger Hilbert space. (At least in a certain sense. It is probably not the case that $L_2(\mathbb{R}^3,\mathbb{C}) \simeq L_2(\mathbb{R},\mathbb{C})^{\otimes 3}$ but it's an okay start)
Ultimately what I think you are looking for is the special relationship between $\hat{x}$, $\hat{y}$, and $\hat{z}$ that justifies grouping them together. The answer is symmetry. The Hilbert space $H = L_2(\mathbb{R}^3,\mathbb{C})$ transforms under rotations of physical space, represented by the group $SO(3)$. These rotations transform states in $H$ into other states in $H$, and they also transform operators on $H$ into other operators on $H$. For example, a rotation can map the $\hat{x}$ operator to the $\hat{y}$ and $\hat{y}$ to $-\hat{x}$, corresponding to a 90 degree rotation of your coordinate system.
When we call $\vec{\hat{x}}$ a vector operator we are really saying that the three operators transform into each other under rotations in this way. But there are still three distinct operators that individually map from $H$ to $H$!
A: Yes, there does exist a rigorous definition of a vector operator, but your intuition is incorrect. Suppose we had a vector operator $\hat{\vec{V}}$ in three-dimensional Euclidean space. The most sensible requirement that we could have for a vector operator is that expectation values of vector operators (which would be a vector of ordinary numbers) should transform as an ordinary vector; that is
\begin{equation}
\langle\psi'|\hat{V}_i|\psi'\rangle=\sum_jR_{ij}\langle\psi|\hat{V}_j|\psi\rangle,
\end{equation}
where $R$ is a rotation which maps the state $|\psi\rangle\rightarrow|\psi'\rangle$. More precisely
\begin{equation}
|\psi'\rangle =\hat{U}(R)|\psi\rangle,
\end{equation}
where $\hat{U}(R)$ is a unitary operator which implements the classical rotation $R$ on the state $|\psi\rangle$ (and may rotate both spatial and spin degrees of freedom). We want the above relation to hold for all states $|\psi\rangle$, which implies that
\begin{equation}
\hat{U}(R)\hat{V}_i\hat{U}(R)^{\dagger}=\sum_j \hat{V}_j R_{ji}
\end{equation}
This is the precise definition of a vector operator. It is a collection of operators which transform as a vector under rotations. Indeed the position operators $\hat{\vec{x}}$ form a vector operator. Analogous definitions hold for scalar operators as well as higher tensor operators (known as irreducible tensor operators). 
As other answers have indicated, though, this does not imply that such operators map the Hilbert space to some sort of expanded space (as your question suggests). It just means that they have prescribed transformation properties under rotations. Much can be learned by studying these transformation properties. You can learn more about irreducible tensor operators and one of the most useful related results known as the Wigner-Eckart Theorem by reading these lecture notes.
