No, that is not enough to say that $B=0$. You must also consider that $$\nabla\times E=-\frac{\partial B}{\partial t}$$ which means that for a magnetic field that is constant spatially but not in time, your conditions would be true but your $B$ field would not be $0$
If, however, we had a case where $\nabla\times E=0$ as well, then (aside from being a very boring situation) we could say that $B=0$. This is done as a simplification tactic. In this case, the $B$ field is constant over space and time. It also means that the $B$ field cannot be affecting any charges because if it were, those charges would accelerate, which would change the $E$ field. Changing the $E$ field means that the curl of $B$ would not be $0$ and $B$ would not be a constant. Thus, since $B=const$, it must not affect charges. Since it does not contribute to $E$ and since it does not affect anything else, we can simplify any calculations by assuming that $B=0$.
The physical meaning of this is that we are letting the background magnetic field that exists everywhere and is unchanging be zero. Similar to the electric potential, we can set the background to be zero and just measure the difference in $B$ between the background and the field of interest. Or, that's at least one way to interpret it.