Noether's current expression in Peskin and Schroeder In the second chapter of Peskin and Schroeder, An Introduction to Quantum Field Theory, it is said that the action is invariant if the Lagrangian density changes by a four-divergence.
But if we calculate any change in Lagrangian density we observe that under the conditions of equation of motion being satisfied, it only changes by a four-divergence term.
If ${\cal L}(x) $ changes to $ {\cal L}(x) + \alpha \partial_\mu J^{\mu} (x) $ then action is invariant. But isn't this only in the case of extremization of action to obtain Euler-Lagrange equations. 
Comparing this to $ \delta {\cal L}$
$$ \alpha \delta {\cal L} = \frac{\partial {\cal L}}{\partial \phi} (\alpha \delta \phi) + \frac{\partial {\cal L}}{\partial \partial_{\mu}\phi} \partial_{\mu}(\alpha \delta \phi) $$
$$= \alpha \partial_\mu \left(\frac{\partial {\cal L}}{\partial \partial_{\mu}\phi} \delta \phi \right) + \alpha \left[ \frac{\partial {\cal L}}{\partial \phi} - \partial_\mu \left(\frac{\partial {\cal L}}{\partial \partial_{\mu}\phi} \right) \right] \delta \phi.    $$ 
Getting the second term to zero assuming application of equations of motion. Doesn't this imply that the noether's current itself is zero, rather than its derivative? That is:
$$J^{\mu} (x) =  \frac{\partial {\cal L}}{\partial \partial_{\mu}\phi} \delta \phi .$$
I add that my doubt is why changing ${\cal L}$ by a four divergence term lead to invariance of action globally when that idea itself was derived while extremizing the action which I assume is a local extremization and not a global one.
 A: Here's what I perceive to be a mathematically and logically precise presentation of the theorem, let me know if this helps.
Mathematical Preliminaries
First let me introduce some precise notation so that we don't encounter any issues with "infinitesimals" etc.  Given a field $\phi$, let $\hat\phi(\alpha, x)$ denote a smooth one-parameter family of fields for which $\hat \phi(0, x) = \phi(x)$.  We call this family a deformation of $\phi$ (in a previous version I called this a "flow").  Then we can define the variation of $\phi$ under this deformation as the first order approximation to the change in $\phi$ as follows:
Definition 1. (Variation of field)
$$
  \delta\phi(x) = \frac{\partial\hat\phi}{\partial\alpha}(0,x)
$$
This definition then implies the following expansion
$$
  \hat\phi(\alpha, x) = \phi(x) + \alpha\delta\phi(x) + \mathcal O(\alpha^2)
$$
which makes contact with the notation in many physics books like Peskin and Schroeder.
Note: In my notation, $\delta\phi$ is NOT an "infinitesimal", it's the coefficient of the parameter $\alpha$ in the first order change in the field under the deformation.  I prefer to write things this way because I find that it leads to a lot less confusion.
Next, we define the variation of the Lagrangian under the deformation as the coefficient of the change in $\mathcal L$ to first order in $\alpha$;
Definition 2. (Variation of Lagrangian density)
$$
  \delta\mathcal L(\phi(x), \partial_\mu\phi(x)) = \frac{\partial}{\partial\alpha}\mathcal L(\hat\phi(\alpha, x), \partial_\mu\hat\phi(\alpha, x))\Big|_{\alpha=0}
$$
Given these definitions, I'll leave it to you to show
Lemma 1.
For any variation of the fields $\phi$, the variation of the Lagrangian density satisfies
\begin{align}
  \delta\mathcal L
&= \left(\frac{\partial \mathcal L}{\partial\phi} - \partial_\mu\frac{\partial\mathcal L}{\partial(\partial_\mu\phi)}\right)\delta\phi + \partial_\mu K^\mu,\qquad  K^\mu = \frac{\partial\mathcal L}{\partial(\partial_\mu\phi)}\delta\phi
\end{align}
You'll need to use (1) The chain rule for partial differentiation, (2) the fact $\delta(\partial_\mu\phi) = \partial_\mu\delta\phi$ which can be proven from the above definition of $\delta\phi$ and (3) the product rule for partial differentiation.
Noether's theorem in steps

*

*Let a particular flow $\hat\phi(\alpha, x)$ be given.


*Assume that for this particular deformation, there exists some vector field $J^\mu\neq K^\mu$ such that
$$
  \delta\mathcal L = \partial_\mu J^\mu
$$


*Notice, that for any field $\phi$ that satisfies the equation of motion, Lemma 1 tells us that
$$
  \delta \mathcal L = \partial_\mu K^\mu
$$


*Define a vector field $j^\mu$ by
$$
  j^\mu = K^\mu - J^\mu
$$


*Notice that for any field $\phi$ satisfying the equations of motion steps 2+ 3 + 4 imply
$$
  \partial_\mu j^\mu = 0
$$
Q.E.D.
Important Notes!!! If you follow the logic carefully, you'll see that $\delta \mathcal L = \partial_\mu K^\mu$ only along the equations of motion. Also, part of the hypothesis of the theorem was that we found a $J^\mu$ that is not equal to $K^\mu$ for which $\delta\mathcal L = \partial_\mu J^\mu$.  This ensures that $j^\mu$ defined in the end is not identically zero!  In order to find such a $J^\mu$, you should not be using the equations of motion.  You should be applying the given deformation to the field and seeing what happens to it to first order in the "deformation parameter" $\alpha$.
Addendum. 2020-07-02 (Free scalar field example.)
A concrete example helps clarify the theorem and the remarks made afterward.  Consider a single real scalar field $\phi:\mathbb R^{1,3}\to\mathbb R$. Let $m\in\mathbb R$ and $\xi\in\mathbb R^{1,3}$, and consider the following Lagrangian density and deformation (often called spacetime translation):
$$
  \mathcal L(\phi, \partial_\mu\phi) = \frac{1}{2}\partial_\mu\phi\partial^\mu\phi - \frac{1}{2}m^2\phi, \qquad \hat\phi(\alpha, x) = \phi(x + \alpha\xi)
$$
Computation using the definition of $\delta\mathcal L$ (plug the deformed field into $\mathcal L$, take the derivative with respect to $\alpha$, and set $\alpha = 0$ at the end) but without ever invoking the equation of motion (Klein-Gordon equation) for the field gives
$$
  \delta \mathcal L = \partial_\mu(\xi^\nu\delta^\mu_\nu \mathcal L), \qquad \frac{\partial\mathcal L}{\partial(\partial_\mu\phi)}\delta\phi = \xi^\nu\partial_\nu\phi\partial^\mu\phi
$$
It follows that
$$
  J^\mu = \xi^\nu\delta^\mu_\nu \mathcal L, \qquad K^\mu = \xi^\nu\partial_\nu\phi\partial^\mu\phi
$$
and therefore
$$
  j^\mu = \xi^\nu(\partial_\nu\phi\partial^\mu\phi -\delta^\mu_\nu\mathcal L)
$$
If e.g. one chooses $\tau > 0$ and sets $\xi = (\tau, 0, 0, 0)$, then the deformation is time translation, and conservation of $j^\mu$ yields conservation of the Hamiltonian density associated with $\mathcal L$ as the reader can check.
Suppose, instead, that in the process of computing $\delta \mathcal L$, one were to further invoke the following equation of motion which is simply the Euler-Lagrange equation for the Lagrangian density $\mathcal L$:
$$
  \partial^\mu\partial_\mu\phi = -m^2\phi,
$$
Then one finds that
$$
  \delta\mathcal L = \partial_\mu(\xi^\nu\partial_\nu\phi\partial_\mu\phi)
$$
so $J^\mu = K^\mu$ and therefore $j^\mu = 0$, which is uninformative.
A: Lagrangian invariant upto a overall 4-divergence and Euler Lagrange equation they together give you $\partial_{\mu}\left(\frac{\partial L}{\partial\partial_{\mu}\phi}\delta\phi\right)=\partial_{\mu}\left(J^{\mu}(x)\right)$
Now if I understood you correctly you are saying essentially if $\dfrac{df}{dx}=\dfrac{dg}{dx}$ then $f=g$ which in general is not true all one can say is $\dfrac{d(f-g)}{dx}=0$ i.e. $f-g=constant$. 
Similarly here $\partial_{\mu}\left(J^{\mu}(x)-\frac{\partial L}{\partial\partial_{\mu}\phi}\delta\phi\right)=0$ would imply 
$j^{\mu}(x)=J^{\mu}(x)-\frac{\partial L}{\partial\partial_{\mu}\phi}\delta\phi$ such that $\partial_{\mu}(j^{\mu}(x))=0$
A: The key point is that the on-shell solutions only extremize the action when boundary conditions are left unchanged, arbitrary transformations on the field in general do not leave the boundary conditions unchanged and this is why the term $\partial_\mu \left(\frac{\partial {\cal L}}{\partial \partial_{\mu}\phi} \delta \phi \right) \neq 0$ since the deformation need not be zero on the boundaries. In the case of boundary conditions at infinity, the deformations need not be regular at infinity and thus can give finite boundary terms.
Also, adding a four-divergence term to the Lagrangian does not leave the action invariant in general, only the physical solutions.
A: First, the differential operation is called "four-divergence" (the four-dimensional divergence), not "fourth divergence".
Second, the action obviously does change under a generic change of the fields i.e. if the change of the Lagrangian is not a four-divergence. It is a completely general functional of the fields so it does change.
Third, the action is stationary when the equations of motion are satisfied. These two conditions are ultimately equivalent. But when deriving the equations of motion, you can't assume that the equations of motion are satisfied. That would be a circular reasoning and you couldn't derive anything.
Fourth, yes, the equations of motion are being used when one derives $\partial_\mu J^\mu=0$ (i.e. action is stationary) but no, the derivation of the Noether's current does not imply that $J^\mu=0$. Your mistake is to confuse what is extremized. The equations of motion only mean $\delta S =0$, not $\delta L=0$ or $\delta{\mathcal L}=0$.
Fifth, your last equation is completely meaningless because the left hand side is finite but the right hand side is infinitesimal. Much like with problems of dimensional analysis (incompatible units), a manipulation with these expressions that obey the basic rules can never end up with a similar mismatch. Your previous "calculation" is also off because you're writing some bizarre expressions that are of second-order. In the variations, $\alpha$ itself is supposed to be infinitesimal, and in valid derivations, there is never a product of $\alpha$ with another infinitesimal quantity such as $\delta \phi$. In effect, your terms are of second-order (doubly infinitesimal) but your analysis doesn't have this higher-order precision so it's wrong.
I think it's a better idea to follow the actual correct derivation instead of your personal attempts to revise the functional calculus that you haven't mastered yet.
