We introduce the $Q$ factor in order to describe how much of the oscillator's energy is lost due to friction in one cycle of oscillations. For it to be of any practical use, the $Q$ factor must be a constant (i.e. time-independent) function of the oscillator's parameters. It should be equally applicable to damped oscillations, as well as steady-state driven oscillations. In the latter case, the energy, as well as the energy loss of the oscillator will additionally depend on the frequency of the external force $\omega$. Since $\omega$ is not the parameter of the system, it has no place in the $Q$ factor. Therefore, for driven oscillations, we calculate the $Q$ factor for resonant frequency $\omega = \omega_0$, where $\omega_0$ is the (undamped) frequency of the oscillator.
Let us first calculate the $Q$ factor for the damped oscillator. Here, the energy of the oscillator $E(t)$ is time dependent (oscillating with decaying amplitude $\sim e^{-t/\tau}$), so the natural definition of the $Q$ factor would be $$Q = 2\pi \frac{E(t)}{E(t) - E(t+T)} = \omega_d \frac{E(t)}{\langle P \rangle (t)}.$$ Here, $T = 2\pi/\omega_d$ is the period and $\omega_d = \sqrt{\omega_0^2 - (1/2\tau)^2}$ is the frequency of damped oscillations. $\langle P \rangle (t)$ is the average power loss due to friction. Now, from the fact that $E(t+T) = E(t) e^{-\frac{2\pi}{\omega_d \tau}}$, we have for the $Q$ factor $$Q = \frac{2\pi}{1 - e^{-\frac{2\pi}{\omega_d \tau}}} \approx \omega_d \tau + \pi + \mathcal{O}\left(\frac{1}{\omega_d \tau}\right)\approx \omega_d \tau,$$ where the approximation holds for very weak damping. It should be noted that the alternative definition $$Q = \omega_d \frac{E(t)}{- \frac{d E}{dt}} = \omega_d \frac{E(t)}{P(t)}$$ does not lead to a time-independent $Q$ factor.
For the steady-state driven oscillator, we have immediately $$Q = \omega_0 \frac{E(t)}{\langle P \rangle (t)} = \omega_0 \frac{E}{\langle P \rangle} = \omega_0 \tau,$$ where we have used the fact that $P = \frac{2}{\tau} E_\text{kin}$ and $\langle E_\text{kin} \rangle = \frac{1}{2} E$ for an oscillator oscillating at its natural frequency.