The term arbitrary parameter, when used in this way, denotes a variable which is introduced at some point of a calculation whose value is not known independently, but which is instead adjusted at the end of the calculation to make the results agree with some external standard.
This is probably best explained by examples. In the situation described by Dyson, he was trying to produce results from a mathematical formalism which, when taken at face value, can only produce infinity after infinity (instead of sensible, finite numbers) because of the presence of divergent integrals which fail to produce any meaningful results. Dyson was employing cut-off procedures which essentially say "I know I'm meant to take this integral $\int_0^\infty \mathrm dx$ all the way to $x\to\infty$, but instead I'm going to stop at $x=X$ and hope that I get something that makes sense". Having done that, you then carry the calculation to its natural conclusion and you get some final result that depends on $X$.
The thing is, though, that there is no natural place $X$ to stop the integral. It is an arbitrary parameter - at the time it is introduced its value could be just about anything - and at the end Dyson takes the value that makes the results of the calculation look as similar as possible to Fermi's experimental results.
The problem with arbitrary parameters of this sort is that if you have too many of them, your theory then becomes too flexible. That is, by appropriate choices of the parameters you can make the results of your theory look like pretty much anythingg, and this then makes the predictive power of your theory somewhat suspect (unless, as Fermi points out, you have a clear physical picture or a precise and self-consistent mathematical formalism to back your theory).
Let me illustrate this a bit more. Suppose that you're Fermi and I'm Dyson, and you have some fresh measurements of $f(t)$ which I reckon have a dependence of the form
You come out of the lab and you show me your results:
I go and crank the numbers, and I say bingo, of course it matches, and my parameters are
$$f(t)=1.28343 \sin(t)-0.147732 \sin(3t)+ 0.0490644\sin(5t)-0.0233293 \sin(7t),$$
and just look what an absolutely gorgeous agreement between theory and experiment:
So what's the problem? Again, my theory is too flexible. If your data had been different, I could have come up with any number of different results:
This is what von Neumann means (or at least the quote Fermi attributes to him), when he says
with four parameters I can fit an elephant, and with five I can make him wiggle his trunk.
Given enough parameters, you can pretty much fit anything. In this example, the original data was obviously linear with a little bit of noise on top (and indeed that's how it was generated), and my sum-of-sines model, while very general, does nothing but cloud the picture: not only does it not take us closer to what's actually going on, it in fact actively sends us in the wrong direction.
This problem is known, in the statistics community, as over-fitting, and it is an important concern to keep in mind whenever a very flexible model is compared against measurements. (For more examples, see this thread at Cross Validated.) In physics, it's applicable if there isn't a clear physical picture to back up the model - but it also happens when there is such a picture, because of course that could be the wrong one!
Let me close with some notes on what happened to fundamental physics after that encounter between Fermi and Dyson. As the full text notes, the meson theory that Dyson was working on turned out to be pretty much wrong; instead, it is a high-level approximation of the more fundamental theory of quarks, and Dyson was probably pushing this approximation past its limits. This quark theory evolved into what is now known as the Standard Model of particle physics.
Now, as it happens, the Standard Model does have a number of what are in essence arbitrary parameters baked in. These include fairly sensible things like the ratio of the proton mass to the electron mass, but they also include some fairly esoteric numbers that seem at first glance to be pretty ad-hoc constructs put in just to save the theory, like the mixing angles of neutrinos. John Baez has counted these arbitrary parameters at 26 in total, which is a long way up from Dyson's four parameters.
However, these parameters have pretty stable values which are used to explain mound after mound of evidence, with more data added everyday at CERN and elsewhere. Moreover, the Standard Model has a strong theoretical underpinning - a precise and self-consistent mathematical formalism - and it comes with a large number of physical perspectives which provide intuition into what's 'really' going on. In these cases, it's really quite OK to have the odd few (dozen) parameters to make things match just right.