What causes the block to slide? That would be gravity.
What tries to prevent this slide? That would be friction.
And friction depends on the normal force $f_k=\mu_k n$.
As you rightfully say, the normal force increases with increasing pushing force $F$, which thus causes increased friction which thus prevents the slide.
Is the surface frictionless, then consider the same case purely from an acceleration component point of view:
With gravity alone, some normal force is present. It balances the perpendicular gravity component, so there is no acceleration in this direction. Only in the sliding direction is the gravity component not balanced.
Now with the push, the normal force increases, so it is far larger than necessary to balance the perpendicular gravity component. So in the perpendicular direction there is now acceleration away from the surface.
But remember that the parallel gravity component is still there and still unbalanced.
Put this parallel and perpendicular component together and the resulting acceleration is straight ahead (in the pushing direction).
In other words, the moving incline is pushing the block forward and upwards, while gravity is pulling it forward and downwards. So certainly, this upwards and downwards tendency can balance and only straight forward acceleration remains