How do you find out the direction of the dipole moment in a charge distribution? For example a sphere with charge density $\rho$ in northern hemisphere and $-\rho$ at southern hemisphere? How can you think about the direction in general?
Generally speaking, the dipole moment of a neutral charge distribution points in a direction that goes from the places with negative charge to the places with positive charge. Moreover, if the distribution has any sort of rotational symmetry axis (either continuous or discrete) then the dipole moment needs to lie along that axis.
To go beyond that, then you need to step away from hand-waving statements and actually calculate the dipole moment through its definition, $$ \mathbf p = \int \mathbf r \, \rho(\mathbf r)\mathrm d\mathbf r, $$ normally through direct and explicit integration.