Unlike rotation, why a $3\times 3$ translation matrix cannot be written in 3D? or can it be? The effect of rotation in 3d on a vector, $\vec{r}=x\hat{x}=y\hat{y}+z\hat{z}$ is given in the form a matrix product:$$\vec{r}\to O\vec{r}$$ where $O$ is a $3\times3$ proper orthogonal matrix. Can we define a $3\times3$ translation matrix $T$ in 3-d so that its action gives $\vec{r}\to\vec{r}+\vec{a}$: $$\vec{r}\to T\vec{r}=\vec{r}+\vec{a}?$$ If yes, what property should $T$ satisfy for example O satisfies $O^TO=OO^T=$identity. I never find it in books. The discussion here gives a $4\times 4$ translation matrix not $3\times 3$.
 A: Matrices represent linear transformations on the space on which they act. 
Translations by a vector $T_\vec{a}\,\vec{x}=\vec{x}+\vec{a}$ don't fall in that class, since
$$ T_\vec{a}(\vec{x}+\vec{y}) \neq T_\vec{a}\,\vec{x} + T_\vec{a}\,\vec{y} $$
A: The translation $\vec{r} + \vec{a}$ in Euclidean 3-space is a trivial shift in the origin that leaves no point in the space invariant. Solving for $T$ in your proposed translation equation (assuming $r$ is invertible) by $T=(r+a) r^{-1}$ gives a non-orthogonal rotation that takes $r$ to the same point as the noted translation.  So as others have said, the answer to your specific question in Euclidean 3-space is "no."
It's worth noting, however, that the $3 \times 3$ proper orthogonal rotation matrices you mentioned live on a manifold that does not fit into Euclidean 3-space.  The conventional sphere in Euclidean 3-space ($S^2$) is not a group manifold, but a coset manifold given by
$$S^2 = SO(3)/SO(2)$$
(see e.g., [1, 273-274, 523-524]).  In other words an $SO(3)$ action on any particular point on $S^2$ results in one of the generators of the rotation quotienting out.  As Zee puts it, the group manifold of $SO(3)$ "is some weird topological space" [1, 273].
So it should not be overly surprising that treatment of non-trivial translations also requires a move to a different topological space.  This can be achieved in three dimensions, just not in three dimensions of Euclidean space.  For example, Coxeter spends almost an entire chapter discussing the concept of Clifford translations in three-dimensional elliptic geometry in [2, Ch.VII].  These translations are constructed as the product of two rotations through equal angles about two absolute polar lines.  This form of translation is quite remarkable, but requires some commitment to understand at an intuitive level.
To bring it back to Euclidean space, left and right Clifford translations can be constructed in Euclidean 4-space.  Coxeter explores this in [3, 140-143] by interpreting the coordinates of a point in Euclidean 4-space as the constituents of a quaternion.  In this context, the product of two rotations
$$x \to axa \\ x \to ax \bar{a}$$
effects a left translation, while the product of the two rotations
$$x \to axa \\ x \to \bar{a}xa$$
effects a right translation.  It may be of interest to note that the product of these left and right translations effects what is usually referred to as an isoclinic rotation in the context of $SO(4)$.
A more robust and general answer to this question that is not restricted to a specific geometry or vector space (to the extent that such an answer exists), would ultimately require delving into transformation groups and adjoint actions which were explored at length by Weyl (e.g. [4, 116-120]), Cartan (e.g. [5, 211-218]), and others.  This would probably go beyond what the OP is looking for, but this is a fascinating area with many open questions.
[1]  Anthony Zee, Group Theory in a Nutshell for Physicists, Princeton University Press, 2016
[2]  HSM Coxeter, Non-Euclidean Geometry, 5th Ed., University of Toronto Press, 1965 (First Published 1942) 
[3]  HSM Coxeter, Quaternions and reflections, The American Mathematical Monthly, vol. 53, no. 3 (1946) http://www.jstor.org/stable/2304897
[4]  Hermann Weyl, The Theory of Groups and Quantum Mechanics, Martino Publishing, 2014 (Translation First Published 1931)
[5]  Élie Cartan, La géométrie des groupes simples, Annali di Matematica, vol. 4 (1927)
