Let's see what we can get out of this. Our derivatives are the 4-vector $\partial_\mu \phi$. The actual most general Lagrangian with your constraints is
\begin{equation}
\mathcal{L} = M^{\mu\nu}(x) \partial_\mu \phi \partial_\nu \phi + A^\mu(x) \partial_\mu \phi + B^\mu(x)\phi\partial_\mu \phi + C(x) \phi^2 + D(x) \phi + F(x)
\end{equation}
First, let's do ourselves a favor. For an action principle, we have that two Lagrangians have the same dynamic if they only differ by a total divergence. If $F$ is well-behaved enough, it can be expressed by a divergence. We can also deal with $A_\mu$ via the integration by parts (The boundary of $\phi$ vanishes at infinity)
\begin{eqnarray}
\int A^\mu(x)\partial_\mu \phi dx &=& - \int \phi \partial_\mu A^\mu(x)
\end{eqnarray}
So that we can combine $A$ and $D$.
You can also use this trick to change the first term to
\begin{eqnarray}
M^{\mu\nu}(x) \partial_\mu \phi \partial_\nu \phi = -\phi \left[(\partial^\mu M) (\partial_\mu \phi) + M(x) \partial_\mu \partial^\mu \phi\right]
\end{eqnarray}
We are left with just
\begin{equation}
\mathcal{L} = M^{\mu\nu}(x) \partial_\mu \phi \partial_\nu \phi + B^\mu(x)\phi\partial_\mu \phi + C(x) \phi^2 + D(x) \phi
\end{equation}
An easy way to see that we can't just transform what remains away is that those are the types of terms we get for a scalar field in a curved spacetime with a source term $D$.
We want our Lagrangian to be Poincaré-invariant. Just considering translation invariance, we have
\begin{eqnarray}
x^\mu &\to& x^\mu + a^\mu
\end{eqnarray}
Our field transforms as
\begin{eqnarray}
\phi(x) \to \phi(x) + a^\mu \partial_\mu \phi(x)
\end{eqnarray}
And the same goes for our various factors. The total variation of the linear part is
\begin{equation}
a^\mu (\partial_\mu D(x)) \phi + D(x) a^\mu (\partial_\mu \phi) + a^\mu (\partial_\mu D(x)) a^\nu (\partial_\nu\phi) = \partial_\mu (a^\mu D (x)\phi) + a^\mu (\partial_\mu D(x)) a^\nu (\partial_\nu\phi)
\end{equation}
We want it to be equal to a total divergence. As this needs to be true for any function $\phi$, this implies that $D$ is a constant. Similarly, Poincaré invariance will constrain $M = \eta^{\mu\nu}$, and $C$ and $B$ are constant as well.
The term in $B$ goes away by
\begin{eqnarray}
\int b^\mu \phi \partial_\mu \phi &=& [\frac{\phi^2}{2}]_\infty = 0
\end{eqnarray}
We just need to get rid of the term linear in $\phi$. For the same reason as $F$, this term can be removed by an appropriate divergence.