Since you have not provided a direct reference, it's hard to be completely sure (and particularly to pin down the details), but there's really only one general idea that this can refer to.
In general, any arbitrary isometry $S$ of euclidean space has the form
$$
\mathbf x\mapsto R\mathbf x+\mathbf t,
$$
where $\mathbf t$ is an arbitrary vector and $R\in\mathrm{O}(3)$ is an orthogonal matrix. The isometry $S$ is then indexed by both these objects, so you can think of $S$ as the ordered pair $(R,\mathbf t)$. It is fairly common for authors to come up with slightly off-beat notations such as $(R|\mathbf t)$ or $\{R,\mathbf t\}$ (example) to emphasize that this is what's happening.
This sort of typographical emphasis is somewhat justified, because the group law in this notation is somewhat complicated: if $S=(R,\mathbf t)$ and $T=(P,\mathbf u)$, then
$$
\mathbf x
\stackrel{S}{\mapsto} R\mathbf x+\mathbf t
\stackrel{T}{\mapsto} P(R\mathbf x+\mathbf t)+\mathbf u
=PR\mathbf x+(P\mathbf t+\mathbf u),
$$
which means that $T\circ S=(R,\mathbf t)\circ(P,\mathbf u)=(PR,P\mathbf t+\mathbf u)$.
Note that the details of the implementation can change, for example, if one translates before the rotation, i.e. $\mathbf x\mapsto R(\mathbf x+\mathbf t)$, but this does not affect the general mechanics. However, it does make it impossible to know exactly what each author means by the notation, which is why the precise notation convention is always specified at the start of each text.
