In order to answer your question, let's follow the derivation of the electromagnetic wave equations in reverse from the final wave equations describing the propagation of electromagnetic waves to the Maxwell equations from which they are derived.
At the end of the derivation one indeed sees how ϵ0 and μ0 end up appearing where one normally expects the square of wave's speed:
\begin{equation}
\nabla^2 {\bf E} = \frac{1}{c^2} \frac{\partial^2 {\bf B}}{\partial^2 t} = \mu_0 \epsilon_0 \frac{\partial^2 {\bf B}}{\partial^2 t}
\end{equation}
\begin{equation}
\nabla^2 {\bf E} = \frac{1}{c^2} \frac{\partial^2 {\bf E}}{\partial^2 t} = \mu_0 \epsilon_0 \frac{\partial^2 {\bf E}}{\partial^2 t}
\end{equation}
This shows how the speed of light in vacuum is related to ϵ0 and μ0.
Going back, the derivation starts from the Maxwell equations, taken here in the absence of free charges and electric currents:
\begin{equation}
\nabla \cdot {\bf E} = 0
\end{equation}
\begin{equation}
\nabla \cdot {\bf B} = 0
\end{equation}
\begin{equation}
\nabla \times {\bf E} = - \frac{\partial {\bf B}}{\partial t}
\end{equation}
\begin{equation}
\nabla \times {\bf B} = \mu_0 \epsilon_0 \frac{\partial {\bf E}}{\partial t}
\end{equation}
(Note that by going from magnetic B field to H field you can make μ0 disappear from the fourth and appear in the third equation.)
This shows that it isn't entirely correct to say that ϵ0 and μ0 are associated with non-time-varying phenomena only: ϵ0 and μ0 do participate in the description of a time-varying phenomenon, namely how the curl of one field depends on the rate of change of the other field.
If you write down Maxwell's equations in their full form without excluding free charges and currents, they'll look like this:
\begin{equation}
\nabla \cdot {\bf E} = \frac{\rho}{\epsilon_0}
\end{equation}
\begin{equation}
\nabla \cdot {\bf B} = 0
\end{equation}
\begin{equation}
\nabla \times {\bf E} = - \frac{\partial {\bf B}}{\partial t}
\end{equation}
\begin{equation}
\nabla \times {\bf B} = \mu_0 {\bf J} + \mu_0 \epsilon_0 \frac{\partial {\bf E}}{\partial t}
\end{equation}
The two extra appearances of ϵ0 and μ0 are not related to electromagnetic waves (as the derivation of the wave equations assumes they're zeroed out) and are actually what you have alluded to in your question, i.e. that the constants are primarily known from electrostatics and magnetism. As you can see they are involved in more than that including time-varying phenomena related to how time variation in one of the two fields generates the other.