You are correct that for a finite potential the probability will not be zero. Inside the 'forbidden zone' the amplitude of the wave function decays with a length-scale $\xi$ given by $$ \xi^{-1} = \sqrt{2m(V-E)}/\hbar $$ so if $V\gg E$ it is very short and the wave-function decays very fast. At the limit of $V\to\infty$ or if the width of the barrier is very long, the probability to find the particle at the other side will be zero. However, for finite barriers the particle can pass through the barrier.

In fact, a very famous phenomenon in physics - the $\alpha$-decay, was explained by Gamow exactly as a process of tunneling through a finite potential. You can read about it here

You are correct that for a finite potential the probability will not be zero. Inside the 'forbidden zone' the amplitude of the wave function decays with a length-scale $\xi$ given by $$ \xi^{-1} = \sqrt{2m(V-E)}/\hbar $$ so if $V\gg E$ it is very short and the wave-function decays very fast. At the limit of $V\to\infty$ or if the width of the barrier is very long (as is implied in the title of your question), the probability to find the particle at the other side will be zero. However, for finite barriers the particle can pass through the barrier.

In fact, a very famous phenomenon in physics - the $\alpha$-decay, was explained by Gamow exactly as a process of tunneling through a finite potential. You can read about it here