We define a symmetry transformation of a system to be any transformation that, when performed, does not change the outcome of a measurement. Wigner's symmetry theorem says that any symmetry of a quantum system is represented by a linear and unitary operator which acts on the Hilbert space of physical states $\mathscr{H}$. So for any symmetry $\mathscr{E}$ there corresponds a unitary transformation $\mathcal{U}(\mathscr{E})$ acting on $\mathscr{H}$.
Suppose $\hat{A}$ is the Hermitian operator corresponding to some observable $A$ which has eigenvalue $\lambda\in\mathbb{R}$ with eigenvector $|\Phi\rangle$. Then, if the system is in a state $|\Psi\rangle$, the probability of measuring the value $\lambda$ of the observable $A$ is given by the Born rule; \begin{align} \text{Prob}(\lambda,\hat{A},\Psi)=\frac{|\langle\Phi|\Psi\rangle|^2}{\langle\Psi|\Psi\rangle\langle\Phi|\Phi\rangle}~. \end{align} We define a symmetry of a quantum system to be one that preserves the above probabilities. However, any unitary operator $\tilde{\mathcal{U}}$ acting on $\mathscr{H}$ will preserve these probabilities; \begin{align} \text{Prob}(\lambda,\hat{A'},\Psi')=\frac{|\langle\tilde{\mathcal{U}}\Phi|\tilde{\mathcal{U}}\Psi\rangle|^2}{\langle\tilde{\mathcal{U}}\Psi|\tilde{\mathcal{U}}\Psi\rangle\langle\tilde{\mathcal{U}}\Phi|\tilde{\mathcal{U}}\Phi\rangle}= \frac{|\langle\Phi|\tilde{\mathcal{U}}^{\dagger}\tilde{\mathcal{U}}\Psi\rangle|^2}{\langle\Psi|\tilde{\mathcal{U}}^{\dagger}\tilde{\mathcal{U}}\Psi\rangle\langle\Phi|\tilde{\mathcal{U}}^{\dagger}\tilde{\mathcal{U}}\Phi\rangle}=\text{Prob}(\lambda,\hat{A},\Psi). \end{align} This would imply that any unitary transformation acting on $\mathscr{H}$ corresponds to a symmetry transformation of the system. I doubt this is true, so where did I go wrong in my definitions and how do I rectify it?