You want to read David Hestenes paper New Tools for Computational Geometry and Rejuvenation of Screw Theory.
In that paper, Hestenes uses geometric algebra to unify linear and angular quantities in mechanics. Linear velocity is merged with angular velocity, linear momentum with angular momentum, and force with torque. This allows both to be handled simultaneously with a single equation of motion.
Your observation on work is most closely related to equation (98) in that paper where the power (rate of change of kinetic energy) is found to be $K=V\cdot \tilde W=\omega \cdot \Gamma+v\cdot f$. $V$ is the generalised velocity (combining linear and angular components), $\tilde W$ is the wrench, the generalised force (combining force and torque), $\omega$ is the angular velocity, $\Gamma$ is the torque, $v$ the velocity, and $f$ is the force. Integrating this over an interval of time, the angular velocity becomes the change in angle, and the linear velocity becomes the displacement. If you integrate the generalised velocity over time you get a screw motion with translation and rotation combined.
The generalised velocity and generalised force are separated into linear and angular components using what Hestenes calls a 'conformal split', where you basically multiply (using the geometric product) both sides of an equation by a constant non-zero bivector. Because you are just multiplying both sides by a non-zero constant, any valid equation will still be valid and the physics is exactly the same. But the split causes a geometric division into components 'in the direction of' the constant (dot product), and components 'perpendicular to' the constant (wedge product). The conformal split in this case is essentially a choice of origin and the point at infinity, and splits the bivector into translational and rotational components.
These 'splits' are how geometric algebra picks a reference frame to work in. As another example, the 'space-time split' is a form of 'projective split' in which you multiply invariant quantities in spacetime by a particular constant time-like vector - the time axis of your observer's inertial reference frame. That turns 4-vectors into a sum of a scalar (component along the time axis) and a 3-component bivector (isomorphic to 3-vectors in a selected reference frame). The time-position, energy-momentum, charge-current and corresponding formulae out of special relativity for 4-force and 4-velocity drop out automatically. The electromagnetic field is a bivector field - applying exactly the same trick turns it into a combination of vector and trivector parts: the electric and magnetic fields. They are inner and outer products of various physical quantities with the constant unit vector along the observer's time axis.
Conformal geometric algebra is the next level up, multiplying both sides by a constant bivector, representing a choice of origin and point at infinity, and it has the benefit of unifying straight lines with circles, planes with spheres, and translations with rotations.
That the torque is the force wedged with the position is actually fudging things slightly. If we apply two equal forces in opposite directions along parallel lines of action, the resultant force-torque is pure torque - we call it a couple. The resultant force is zero, and the position is undefined, so their wedge is not well-defined either, but the torque most definitely is.
We can understand this better with a bit of projective geometry. We add an extra dimension, so we have coordinates $(x,y,z,w)$ and embed the universe along the $w=1$ hyperplane. (So all our real-world points have coordinates of the form $(x,y,z,1)$.) Our force-torque bivector is represented by a plane through the origin $(0,0,0,0)$ which might or might not intersect the $w=1$ plane. If it does, this is the line of action of the force. If the force-torque plane is parallel to the $w=1$ plane, there is no intersection, no line of action, and we have a pure couple.
In the case where the bivector intersects the plane, the component in the $w$ direction is the force, and the component perpendicular to the $w$ direction is the torque. If the plane cuts $w=1$ a distance $d$ from the real-world origin $(0,0,0,1)$, and the force component in the $w$ direction is $f$, then the torque component in the direction perpendicular to $w$ is $df$. This is why we multiply the force $f$ by the distance to the line of action $d$ to get the torque. It's really doing things backwards. If the torque and force components are known, then the distance to the line of action is torque/force. If force is zero, this makes no sense and there is no line of action.
Geometric algebra is certainly beautiful, if you like finding deep connections, and want an intuitive geometric understanding of some otherwise very abstract and opaque concepts. But it is quite hard to find basic introductory explanations for its more advanced ideas.