Exact proof that 3 isolated mass points forming a non-degenerate triangle and having total momentum and angular momentum $0$, remain on the same plane Let's suppose each of the mass points has a unit mass. Then, mathematically, the problem is the following.
Let $A$, $B$, $C$ be differentiable function $\mathbb R\to \mathbb R^3:t\mapsto A(t),B(t),C(t)$ with the following properties.

*

*$v_A+v_B+v_C=0$ where $v_A:=\dot A$, $v_B:=\dot B$, $v_C:=\dot B$ (momentum is constant $0$)

*$A \times v_A+ B\times v_B+ C\times v_C = 0$ (angular momentum is constant $0$)

*$(C-A)\times (C-B)$ nowhere vanishes. (the $ABC$ triangle is never degenerated)

from 1, $A+B+C$ is constant. Without loss of generality, suppose it is $0$, that is,
$$A+B+C=0 \tag{4}$$
I can prove that under these conditions, the 3 points remain forever in the initial plane of the $ABC$ triangle (i.e. $(C(t)-A(t))\times (C(t)-B(t))$ is collinear with $(C(0)-A(0))\times (C(0)-B(0))$ for any $t$). But I feel my proof is too long and complicated and I'd like to see a more elegant proof.
My proof consists of two steps. In the first step, I prove  that for every $t$, $v_A(t)$, $v_B(t)$ and $v_C(t)$ are in the 2-dimensional subspace $P(t)$ spanned by $A(t)$ and $B(t)$. In the second step, I prove that the normal vector $n(t)$ to $P(t)$ is constant.

Step 1. By condition (4), $P(t)$ is a 2-dimensional linear subspace of $\mathbb R^3$. Let $n$ be a unit normal vector to $P(t)$. Let $v'_A$, $v'_B$ and $v'_C$ be real numbers such that $v'_An$, $v'_Bn$ and $v'_Cn$ are the components of $v_A(t)$, $v_B(t)$ and $v_C(t)$ orthogonal to $P(t)$. Substituting $v_C = - (v_A + v_B)$ from 1. into 2, 
 
$$0 = A \times v_A + B \times v_B - C \times (v_A + v_B )  =  (A - C) \times v_A + (B - C) \times v_B \tag{5}$$
For the normal component of the vector on the right side,
$$(A - C) \times v'_An + (B - C) \times v'_Bn = (v'_A(A - C) + v'_B(B - C)) \times n \tag{6}$$
Since the vector $v'_A(A - C) + v'_B(B - C)$ is orthogonal to $n$ and $n \neq 0$, from (6) follows $$ (A - C) v'_A+ (B - C) v'_B = 0.$$
Thus, by the linear independency of $(A -C)$ and $(B - C)$, $v'_A = v'_B = 0$ and this completes step 1.

My proof of step 2. isn't exact enough so I even don't write it down here. Could somebody give a mathematically exact proof for step 2, or for the whole statement?
 A: Here is a short self-contained proof - inspired by the earlier remarks and answers.
By the first condition, the baricenter $(A+B+C)/3$ is constant. The assumptions and the conclusion are invariant under shifting $A$, $B$, $C$ by a constant vector, hence we can assume from the beginning that $A+B+C=0$. Then the first condition is automatic, while the second and third conditions can be rewritten as (after substituting $C=-A-B$):
$$(A+2B)\times v_B=v_A\times(2A+B),$$
where $A$ and $B$ are linearly independent. Let us call $X$ the common two sides of this equation. First, $X$ is orthogonal to $A+2B$ and $2A+B$, hence it is also orthogonal to $A$ and $B$. Second, decomposing $X$ as
$$X=A\times v_B+2B\times v_B\quad\text{and}\quad
X=2 v_A\times A+v_A\times B,$$
and observing that any cross product containing $A$ (resp. $B$) is trivially orthogonal to $A$ (resp. $B$), we infer that both terms on both right-hand sides are orthogonal to $A$ and $B$. In particular, the derivative of $A\times B$ is orthogonal to $A$ and $B$, whence $A\times B$ is constrained to a ray (half-line starting at the origin). Done.
A: Let us denote positions as $X_i$ and velocities as $v_i$, where $i = A,B,C$. A normal vector to the plane of $X_i$ is
$$ n = (X_B-X_A) \times (X_C-X_A) = X_B\times X_C - X_B\times X_A - X_A\times X_C$$
and its time derivative is
\begin{align} 
\dot{n} & = v_B\times X_C + X_B \times v_C - v_B \times X_A - X_B\times v_A - v_A \times X_C - X_A\times v_C \\
& = v_B \times (X_C-X_A) + v_C\times (X_A-X_B) + v_A\times (X_B-X_C) \\
\end{align}
and since you've already shown that the $v_i$ lie in the plane of the points $A,B,C$, we have that $v_i\times (X_i-X_j)$ is normal to this plane, i.e. $\dot{n}$ is parallel to $n$ at all times, and hence the normal direction is constant - only the length of $n$ changes.
A: My proof has something in common with that of ACuriousMind.
Let's consider vectors
$$
n = (C-A)\times(C-B) = A\times B + B\times C + C\times A \quad (1)
$$
and
$$
l = \frac{n}{\sqrt{n\cdot n}}. \quad (2)
$$
From properties 1 and 2 and equality $A\cdot(B\times C) = B\cdot (C\times A)$, we derive following equalities:
$$
v_A \cdot n = v_B \cdot n = v_C \cdot n = 0\quad (3)
$$
Indeed,
$$
v_A \cdot n = v_A \cdot (A\times B) + v_A \cdot (B \times C) + v_A \cdot (C\times  A) = 
$$
$$
= -(v_B+v_C)\cdot(B\times C) + B\cdot (v_A\times A) + C\cdot (A\times v_A) = 
$$
$$
= -(v_B+v_C)\cdot(B\times C) - B\cdot (v_B\times B + v_С\times C) + 
C\cdot (v_B\times B + v_С\times C) = 
$$
$$
= -(v_B+v_C)\cdot(B\times C) - v_C\cdot(C\times B) + v_B\cdot(B\times C) = 0.
$$
From (3) and $A\times (B\times C) = B(A\cdot C) - C(A\cdot B)$ follow equalities
$$
(v_A\times(B-C))\times n = (v_B\times(C-A))\times n = (v_C\times(A-B))\times n = 0\quad (4).
$$
Indeed,
$$
(v_A\times(B-C))\times n = (v_A\times(B-C))\times((C-A)\times(C-B)) =
$$
$$
= (C-A)\left((v_A\times(B-C))\cdot(C-B)\right) - (C-B)\left((v_A\times(B-C))\cdot(C-A)\right) = 
$$
$$
= -(C-B)\left(v_A\cdot((B-C)\times(C-A))\right) = -(C-B)(v_A\cdot n) = 0.
$$
From (4) and
$$
\dot{n} = v_A\times(B-C) + v_B\times(C-A) + v_C\times(A-B)
$$
follows
$$
\dot{n} \times n = 0. \quad (5)
$$
Now we can derive that vector $l$ (2) doesn't depend on $t$:
$$
\dot{l} = \frac{\dot{n}}{\sqrt{n\cdot n}} - \frac{n(\dot{n}\cdot n)}{(n\cdot n)^{3/2}} = \frac{n\times(\dot{n}\times n)}{(n\cdot n)^{3/2}} = 0\quad (6)
$$
From equalities (1-3,6), it follows
$$
\frac{d}{dt} (A\cdot l) = v_A \cdot l = 0. \quad (7)
$$
For vector $n$ (1), we have
$$
A\cdot n = A\cdot(B\times C) = B\cdot(C\times A) = B\cdot n = C\cdot n \quad (8)
$$
Hence, (7) also means
$$
\frac{d}{dt} (B\cdot l) = \frac{d}{dt} (C\cdot l) = 0\quad (7').
$$
Equalities (6), (7), (7'), and (8) are equivalent to the statement that three masses always remain on the same plain.
A: Proof for the whole statement:
We can assume that 0 is the baricenter, A + B + C = 0, and A,B,C are pairwise independent.
Step 1: $(\forall t)$, $v_A,v_B,v_C$ lie in the plane $P :=A B C.$
(From an another answer.) From property (2):
$$A\times v_A + B\times v_B = -(A+B)\times (v_A + v_B)$$ which can be rearranged as
$$ (2A + B)\times v_A = (-2B - A) \times v_B =: X. $$
X is orthogonal to (2A + B) and (-2B - A) both, they span P, therefore X is orthogonal to P, so $v_A$ and $v_B,v_C$ lie in P. This concludes step 1.
Step 2: $(\forall t),$ the derivative of the normal vector of P is 0.
Let n denote the normal vector of P, $n:=A\times B/||A\times B||$.
$$(A \times B)' = A'\times B + A\times B' = v_A \times B + A \times v_B, $$
so $(A \times B)'$ is parallel to n. This gives
$$ n' = ( A\times B/||A\times B|| )' = (A\times B)' \cdot (1/||A\times B||) + (A\times B) \cdot (1/||A\times B||)' $$
which means $n'$ is parallel to n as well, and considering n being a unit vector, this means $n'= 0$. This concludes step 2.
Step 2 implies the statement.
