As developed in most physics mechanics texts for the motion of a system of particles (for which a rigid body is a special case), you use an inertial frame with a fixed origin, but segregate the angular momentum into two parts: that of the center of mass moving with respect to the fixed origin and that of the body moving with respect to the center of mass, then equate the change in both terms (the overall change in angular momentum) to the net external torque. You are not using a non-inertial (accelerating frame) attached to the moving center of mass, it just seems that way because if you take the center of mass as the reference point the net external torque equals the change in angular momentum of the body about the center of mass because the angular momentum of the center of mass is zero with the center of mass as the reference point. If you take a point other than the center of mass as the reference point, you have another term for the change in angular momentum due to torque to consider. The answer by @Kashmiri shows the mathematical details. An older physics text, Mechanics by Symon, also provides the mathematical details with clarity. Note that these results are true for a system of particles in general, for which a rigid body is a special case.
It is possible to use a non-inertial reference frame, but torques from fictitious forces must also be considered for an arbitrary reference point and arbitrary motion (rotation and translation) of the origin. The Symon text, Mechanics, has a problem asking that you generalizedevelop the evaluationrelationship for torque and change in angular momentum using an origin that is accelerating where the fictitious torques must be considered.
This question is also asked on Man on a railroad car on this exchange. You may find my answer there helpful; it is basically the same answer as @Kashmiri provides here.