As long as the electromagnetic field is free, your approach works fine. But as soon as you want to include interactions, things become more complicated. For one thing, the term that couples the electromagnetic field to the rest of fields is
where $j_\mu$ is the electric current of such fields.
You cannot write this term directly in terms of $\boldsymbol E,\boldsymbol B$, so you must work with $A^\mu$. That's pretty much it.
This is even more drastic in the case of Yang-Mills, which is a self-interacting version of electromagnetism. Here, even in the absence of an external current $j_\mu$, and due to the self-interactions, it is impossible to formulate the theory directly in terms of $\boldsymbol E,\boldsymbol B$. Here the vector potential $A^\mu$ is unavoidable.