In a circular motion, like the horse rides in a merry-go-round, there is only one force that acting on it, the centripetal force that pulls the ride inward. I'm struggling to answer why the horse rides swing outward instead?
The only REAL force is the centripetal force. A merry-go-round is a non-inertial frame of reference and there are inertial forces present in it too. The centrifugal force, The Euler force and the Coriolis force.
The first one for example, from an observer's point of view, who is sitting in the middle of the merry-go-round pushes the object (Horse in this case) far from the center.
As the ride starts, The Euler force will be the apparent force pushing the person to the back of the horse, and as the ride comes to a stop, it will be the apparent force pushing the person towards the front of the horse.
You can read about these forces and the Coriolis force, which is a major player in the earth climate and weather changes, in the provided links or analytical mechanics books like "Classical Dynamics of Particles and Systems" by Thornton and Marion. Or Fowles analytical mechanics.
As you clearly understand, the horses need an inward centripetal force to undergo circular motion. But they can't get this if they are hanging vertically, or tilting inwards. Draw a free body diagram, and it's a good idea to have the horse swinging on an eye or loop that cannot transmit torque when you do this. Either this, or simply hang them on ideal strings that can support no transverse forces.
In either of these cases there are only two forces: gravity, and a force (tension) directed along the tether.
The only one of these two that can provide the necessary inwards force (the horizontal component) is the tension. And the string / tether must be such that it tilts inwards, otherwise it cannot give this force. The horses must therefore tilt outwards.
Centripetal force is not the only force acting on the horse. In fact it is not a distinct force at all. It is the resultant of other real forces acting on the horse. This resultant causes centripetal acceleration, ie circular motion.
The diagram shows the horse (grey block) fixed on a pole which is suspended at an angle $\theta$ to the vertical as it rotates on the merry-go-round. The real forces acting on the horse are its weight $W$ and the tension $T$ in the pole. The vector sum of these 2 forces is $F$. It is responsible for the centripetal acceleration $a$, so it is called the centripetal force $F=Ma$.
The horse is not accelerating vertically so $T\cos\theta=W$. The horse is accelerating horizontally to the left, towards the centre of the merry-go-round. Since $T$ is the only force with a horizontal component, the resultant force causing centripetal acceleration is $F=T\sin\theta=W\tan\theta$.
If the pole were vertical $(\theta=0)$ there would be no horizontal component of force $T$ so the resultant force would be $F=0$. Without any horizontal force the horse could not move in a horizontal circle. As the merry-go-round gets faster the horse tilts further outwards ($\theta$ increasing) to provide a larger centripetal force $F=W\tan\theta$.