Force could be treated as a vector even if mass behaved different in different directions. In this case, mass would not be a simple scalar, but it would be what is called a second rank tensor. Treating vectors as column $3 \times 1$ matrices, force and acceleration would be
$
\mathcal{F}=\left(
\begin{matrix}
F_x\\
F_y\\
F_z\\
\end{matrix}
\right)
\quad
\text{and}
\quad
\mathcal{A}=\left(
\begin{matrix}
a_x\\
a_y\\
a_z\\
\end{matrix}
\right);
$
the second rank tensor representing mass would be simply a $3 \times 3$ matrix:
$
\mathcal{M}=
\left(
\begin{matrix}
m_{xx} & m_{xy} & m_{xz}\\
m_{yx} & m_{yy} & m_{yz}\\
m_{zx} & m_{zy} & m_{zz}\\
\end{matrix}
\right);
$
and Newton's second law would read deceptively like its well known form, $\mathcal{F}=\mathcal{M}\mathcal{A}$, but definitely not with the same content:
$
\left(
\begin{matrix}
F_x\\
F_y\\
F_z\\
\end{matrix}
\right)
=
\left(
\begin{matrix}
m_{xx} & m_{xy} & m_{xz}\\
m_{yx} & m_{yy} & m_{yz}\\
m_{zx} & m_{zy} & m_{zz}\\
\end{matrix}
\right)
\left(
\begin{matrix}
a_x\\
a_y\\
a_z\\
\end{matrix}
\right)
\quad
\text{or}
\quad
\begin{matrix}
F_x = m_{xx}a_x+m_{xy}a_y+m_{xz}a_z,\\
F_y = m_{yx}a_x+m_{yy}a_y+m_{yz}a_z,\\
F_z = m_{zx}a_x+m_{zy}a_y+m_{zz}a_z.
\end{matrix}
$
According to this general formulation, the resistance a body has against a change on its state of motion is direction-dependent. In particular, if a force acts in one direction only, the body may accelerate in different directions as well depending on which elements of the mass matrix are nonvanishing.
In some sense, the above relations are similar to the rotational dynamics of a rigid body, where torque relates to angular acceleration by means of a "rotational inertia" matrix. The consequences in such case are less abstract. If a body is set to spin along some direction, its spinning direction usually deviates from the original direction in "unexpected" ways depending on the rotational inertia matrix of the body, initial conditions and applied torque. For instance, a spinning top will precess due to the action of gravity, but this effect is much more impressive as seen here at 35:20; it will precess even if gravitational torque is absent if it is set to spin along a direction other than its symmetry axis, like here at 0:54; for a very dramatic situation of instabilities on the free rotation, see here (starting at 0:28).
Nevertheless, the two cases may be very different in general. The most general rotational inertia matrix has the very important feature of being symmetric (it is the same as its transpose) what guarantees it can be diagonalized so that torque and angular acceleration can be made parallel in some well chosen coordinate system. On the other hand, the most general mass matrix may not have such property (not that I can tell at least). Because of that, it is possible, for some values of the elements of the mass matrix, that a force along a single direction acting on a body's center of mass produces an acceleration in some perpendicular direction only.
In the context of classical mechanics, as far as I can see, experiments dealing only with translations (forces applied at the center of mass of the body) may not be sufficient to determine all nine elements of its mass matrix, requiring experiments dealing with rotations of the body as well.
In the context of quantum mechanics, as far as now (see this Wikipedia page), experiments indicate the mass matrix for a single body is diagonal with all mass elements equal, so mass can be taken as a scalar quantity.