People sometimes talk about on-shell symmetries: symmetries of the equations of motion or the S-matrix, which do not hold off-shell (i.e. at the level of the action, path integral, correlators, etc). Electric-magnetic duality transformations $$ \begin{aligned} F &\rightarrow \cos(\theta)\,F - \sin(\theta)\,\star F \\ \star F &\rightarrow \sin(\theta)\,F + \cos(\theta)\,\star F \end{aligned} $$ are an example of this. These transformations leave the equations of motion and the S-matrix invariant (with a corresponding selection rule), so they would qualify as a symmetry.
However, this symmetry is usually broken by non-perturbative effects (e.g. monopoles), to which the usual S-matrix is insensitive. In addition, it is well known that dualities are not symmetries of the full theory but instead a better understood as a change in our description of the physics. For instance duality transformations relate the partition function of the theory at different values of the coupling (see this paper for an example).