I have a question on the invariance of the action under symmetry transformation. As the simplest example, here I consider two dimensional Harmonic oscillator. After some rescaling, the Hamiltonian can be written as $H=\frac{1}{2}(x^2+y^2+p_x^2+p_y^2)$. This model is known to have more symmetries than just the spatial rotation and the time translation.
The spatial rotation is defined by $$x'=x\cos\theta -y\sin\theta, y'=y\cos\theta+x\sin\theta, p_x'=p_x\cos\theta-p_y\sin\theta, p_y'=p_y\cos\theta+ p_x\sin\theta,$$ which is generated by $L=xp_y-yp_x$.
Instead, here I want to consider a transformation $$x'=x\cos\theta+p_y\sin\theta, y'=y\cos\theta+p_x\sin\theta, p_x'=p_x\cos\theta- y\sin\theta, p_y'=p_y\cos\theta- x\sin\theta,$$ which is generated by $Q=p_xp_y+xy$. This is a canonical transformation since it preserves the poisson bracket, and it is a symmetry of the system since the Hamiltonian remains unchanged, i.e., $H(x',y',p_x',p_y')=H(x,y,p_x,p_y)$.
My question is about the invariance of the action $$S[x,y,p_x,p_y]=\int dt\Big(p_x\dot{x}+p_y\dot{y}-H(x,y,p_x,p_y)\Big).$$ Under the spatial rotation, $S[x',y',p_x',p_y']=S[x,y,p_x,p_y]$ holds. However, under the symmetry generated by $Q$ above,
$$S[x’,y’,p_x’,p_y’]\neq S[x,y,p_x,p_y]$$ because $$\big(p_x’\dot{x}’+p_y’\dot{y}’\big)-\big(p_x\dot{x}+p_y\dot{y}\big)=\frac{d}{dt}\Lambda(x,y,p_x,p_y)$$ where $$\Lambda(x,y,p_x,p_y)=\sin\theta\cos\theta(p_xp_y-xy)-\sin^2\theta(xp_x+yp_y).$$ So, although the Hamiltonian is unchanged, the action is changed by a surface term that explicitly depends on not only $x,y$ but also $p_x,p_y$. Usually when one considers variations of the action in Hamiltonian formalism, one fixes the initial and the final values of $x,y$ only and leaves the initial and the final values of $p_x,p_y$ unfixed. Hence, the action is changed more than a constant. I thought the invariance of the action is of fundamental importance. Why is this change acceptable? Is there any consequence resulting from it?
Actually the same issue can be discussed in the Lagrangian formalism as well (the surface term depends on $\dot{x},\dot{y}$ but still called a symmetry). But to avoid complication here I am discussing in the Hamiltonian formalism.
[added on Sep 15:] To clarify my point, I understand that a conservation of $Q$ can be checked simply by computing the Poisson bracket with $\{Q,H\}=0$. In the Lagrangian formalism,this corresponds to a "Classical mechanics viewpoint" discussed in this post.
My question is different. I am asking why this can be understood as a symmetry even although the action is changed more than a constant.