Let us assume that when you shorten the pendulum you are not doing any work on the bob.We know that that total energy of a pendulum is equal to the potential energy possessed by it at the highest point.Also the Maximum height reached by the pendulum is $l(1-cos\theta)$ where l is effective length and $\theta$ is the angular amplitude. So total mechanical energy of pendulum can be written as $mgl(1-cos\theta)$ . So when you reduce l, $ \theta$ increases such that $cos\theta$ decreases and $1-cos\theta$ increases keeping total mechanical energy same.So when you reduce l Angular amplitude increases.But you cannot reduce l indefinitely because a point will be reached when angular amplitude will be as high as $\pi$ which is the maximum possible angular amplitude. Let this limiting value of l be $l_{limiting}$.
Now the question arises : What happens when you reduce effective length such that it becomes less than $l_{limiting}$.Well in that case the total mechanical energy won't change (since we haven't done any work on the bob) but potential energy at the maximum height i.e at angular displacement of $\pi$ with respect to vertical will be less than Total mechanical energy so the body will have some kinetic energy at this point i.e It will execute motion in vertical circle.(Refer to diagrams provided);
Here's the math: initial Mechanical energy = $mgl_1(1-cos\theta)$
Final mechanical energy = $mgl_2(1-cos\phi)$
But since work done = 0 , Initial mechanical energy = Final mechanical energy
$mgl_1(1-cos\theta)$ = $mgl_2(1-cos\phi)$
where $l_1$ is initial length , $l_2$ is final length , $\theta$ is initial angular amplitude and $\phi$ is final angular amplitude .
Solving this equation we can calculate the value of $\phi$.
If $l_2$ < $l_1$ then $\phi$ > $\theta$
Let me know if this was clear enough.
