I understand the right-hand rule and direction of the force on an arm, but what exactly is this 'direction' that results from the cross product of the vectors $T = r \times F$? On paper, what is this torque 'direction' that comes in or out of a page? What does the torque 'direction' do on a door that is being opened?
There are probably lots of duplicates, so my apologies, but for clarity I will try a short answer, as the graphic from Wikipedia is particularly illustrative.
The torque is perpendicular, ( orthogonal) to the other two vectors, so it could be the line where the hinges are located, depending on the direction of the other two forces.
From Wikipedia Torque
Torque, moment, or moment of force (see the terminology below) is the tendency of a force to rotate an object about an axis,1 fulcrum, or pivot. Just as a force is a push or a pull, a torque can be thought of as a twist to an object. Mathematically, torque is defined as the cross product of the vector by which the force's application point is offset relative to the fixed suspension point (distance vector) and the force vector, which tends to produce rotation.
Loosely speaking, torque is a measure of the turning force on an object such as a bolt or a flywheel. For example, pushing or pulling the handle of a wrench connected to a nut or bolt produces a torque (turning force) that loosens or tightens the nut or bolt.
In short, you are to think of the direction of the torque as pointing along the axis of the rotation it would induce in a rigid body initially at rest.
But if the conception of torque as a vector out of the page seems artificial, that's because it is.
Torque is not fundamentally a quantity that is a vector but a directed plane or directed area. Such an object is called a bivector. When we speak of torque as a vector we're using a non-general definition that works because we happen to live in three dimensions.
Rotations fundamentally transform planes, rather than leaving axes invariant. In a general number of dimensions, you specify a rotation (more often known as a proper orthogonal transformation in such a context) by specifying its action on two-dimensional, linearly independent subspaces. The simplest possible rotation acts on one plane, but orthogonal transformations can act on many planar subspaces at once.
Torques, producing rotations, are also, fundamentally, bivectors.
In three dimensions, there is at most one plane that can be acted on in this way, so we can cheat a bit and define the rotation through its axis, because in three dimensions because one can define a plane by the unit normal vector to it. But in four dimensions, you cannot define a plane by a unit normal vector. The orthogonal complement of a plane in four dimensions is another plane, so that the axis notion in four dimensions is meaningless - it won't define a rotation and could not define a torque if we lived in a four spatial dimensional universe and had occasion to calculate moments of forces there.
Another notion you may meet in the future is the Hodge Dual. This is a generalization of the defining a plane through an axis in three dimensions, as is done with torques and rotations.