For completeness let us write down the potential
$$V(\phi)=-\mu^2\phi^2+\lambda \phi^4$$
In what follows we will use the term "invariant", which means: "does not change".
The potential (as well as the Lagrangian) is invariant under the symmetry transformation
$$\phi \rightarrow \phi e^{i\theta}$$
This symmetry of the potential is intact in the broken and in the unbroken phase for all values of $\phi$.
You can see this directly from the graphs: the symmetry transformation is a rotation of $\phi$ around the origin and both graphs are symmetric under
the rotation around the z-axis, which goes through the origin of the complex $\phi$-plane, which I will call just origin in what follows.
In general a rotation of $\phi$ around any point $\phi_1$ can be written as
$$\phi = \phi_1 + \phi_2\rightarrow \phi_1 + \phi_2 e^{i\theta}$$
which will turn into the transformation above when identifying $\phi_1=0$ with the origin. For $\phi_1 \neq 0$ the potential is not invariant under the
transformation in the broken as well as in the unbroken phase. Also
this can be seen directly from both graphs as they have only the z-axis as symmetry axis.
The point is, that we are interested in fluctuations around the ground state. In the unbroken phase the ground state
is at the minimum of the potential, so it is identical to the origin. Since the potential is symmetric under rotations around the z-axis,
the potential is symmetric under rotations of $\phi$ around the ground state.
In the broken phase the ground state is one of the $\phi$-values of the minimum-circle. In particular this is not the origin.
Since we have already stated that the potential is not symmetric under rotations of $\phi$ around any other $\phi_1$ than the origin,
the potential is not symmetric under rotations of $\phi$ around the ground state.
To summarize: The potential is invariant under rotations of $\phi$ around the ground state in the unbroken phase,
but it is not invariant under rotations of $\phi$ around the ground state in the broken phase.
I think this is meant when we say: the ground state breaks the symmetry.
Note that the above symmetry statements for the potential are also true for the Lagrangian
$$\mathcal{L} = \partial^\mu \phi \partial_\mu \phi - V(\phi)$$