What method should I use to find the best fit curve that satisfies the model constraints (quadratic at low x, linear at large x, anything goes for intermediate x) and fits the data as best as possible?
It depends on what your goal exactly is, whether you want to find $\alpha,\beta$ as accurately as possible or whether you want "good fit" in the sense that the curve passes as close as possible to the points measured. It also depends on how much do you know about error of $x$ and $y$ values.
The simplest thing that would satisfy your requirements would be to introduce a parametrized function of $x$ (with parameters being $\alpha, \beta$ and possibly others as well) that would obey mentioned asymptotic behaviours and find the parameters of the function with the least-squares method.
https://en.wikipedia.org/wiki/Least_squares
You may, for example, assume there are three measurement-characterizing values of $x$:
- $\lambda_\alpha$, beyond which quadratic term becomes non-dominant;
- $\lambda_\beta$, beyond which linear term becomes dominant;
- $w_\beta$, interval of $x$ where linear term gains weight.
Then you can use a function like
$$
f(x) = \alpha x^2 e^{-x/\lambda_\alpha}+ \beta x \frac{1}{1+e^{-(x/w_\beta-\lambda_\beta/x)}}
$$
and fit all parameters with least-squares method.