Torque is defined as $\vec\tau = \vec r \times \vec F$, where $\vec r$ if what you call the "lever arm." This displacement vector (the lever arm) points from the axis of rotation to where the force is applied. This means that torque does not even makes sense as a concept without a chosen axis of rotation and the resulting lever arm. In fact, torque is in no sense absolute: the same force can create different torques depending on where the axis of rotation is chosen to be.
If there is a torque applied to an optical apparatus due to changing the angular momentum of some light, then how might an experimenter measure such a torque? He or she might attach said optical apparatus to a device that is free to rotate against some force gauge which has been calibrated to measure foot-pounds (or some other torque unit) within this apparatus. The lever arm for that applied torque would then be the distance from where the light hits the apparatus to the axis of rotation. Obviously, there would be some experimental error and the measurement would likely be more subtle than this, but the upshot is that if torque is being measured it only makes sense to say that it is measured with respect to some axis of rotation.