The Direction of Magnetic Force Between Magnets Consider the experiment of a south pole of one bar magnet attracting the north pole of another bar magnet. Now, the magnetic field of the south end of the bar magnet points towards that end. If the north end of the other magnet is attracted to the south end of the other, that means the force is directed along the field line. But the magnetic force should always be perpendicular to the field direction, so this seems like an inconsistency.
 A: Strangely this is often not explained in beginning classes. You are probably thinking about the Lorentz force law
$$
F = q v \times B
$$
in which case the force is indeed always perpendicular to the direction of the magnetic field. However, this is not the full story. There's also the force that a dipole feels, which is given by
$$
F = q v \times B + \nabla ( \mu \cdot B)
$$
where $\mu$ is the magnetic moment vector. All the little spins in the bar magnet have their dipoles aligned. Because the $B$ field of the other magnet is getting weaker with distance, and $B$ is changing, then $\nabla ( \mu \cdot B)$ is changing. This is the force responsible for two bar magnets attracting.
A very simple way to derive this formula is to note that the potential energy of a magnetic moment in an external magnetic field is
$$
U = - \mu \cdot B
$$
at which point you use
$$
F = - \nabla U = \nabla (\mu \cdot B).
$$
For more information: https://en.wikipedia.org/wiki/Force_between_magnets
Edit: OP has asked in the comments

Aren't all magnetic force formulas derived from the Lorentz formula? A dipole is just a loop of current, so we should sum the Lorentz force law over the loop.

This is correct. You can derive the force law from considering the Lorentz force on a small loop of current. The simplest way is to have the current loop be a tiny square and Taylor expand $B$ at different segments of the wire. You will also need to use the equation $\nabla \cdot B = 0 $ to get the final expression. Indeed, if you draw a little diagram, you see that if the magnetic field strength is changing, then the field lines "splay" out:

Therefore, at every point, the force really is perpendicular to both the direction of the current and the magnetic field. However, you must sum together all the individual components to get the net force.
Let me now anticipate the next question: "But wait, how can the dipole force do work if the Lorentz force law cannot do work?" There is a good discussion of this in Griffiths. One solution is to just say that the magnetic force never does work, except on elementary dipoles like electrons. This is correct, but maybe doesn't fully answer the question. What if you really did just have a tiny current loop?
Well, looking at the diagram above, imagine if the current loop had some downward vertical velocity. Then the Lorentz Force law will actually point in a direction at each point along the wire to slow down the current, and hence weaken the dipole, hence not changing $U = - \mu \cdot B$ while $B$ is actually changing. The "work," therefore, come from whatever is keeping the current going through the wire constant! You can maybe imagine that there is some sort of perfect battery keeping the current flowing through the battery, that automatically adjusts to keep the current $I$ exactly equal, even as the tiny loop travels throughout the background magnetic field. The work done that changes $U = - \mu \cdot B$ is therefore all done by the battery in this case. (It then appears that maybe elementary dipoles are a bit like these current loops with magical batteries that keep their magnitude of spin always constant, and it is this magical battery that does all the work...?)
