First, quite generally, it is not true that the action $S$ is ever required to be a multiple of Planck's constant $h$. What quantum mechanics implies is pretty much exactly the opposite thing. The formulation of quantum mechanics (any quantum mechanical theory) that uses the action is the Feynman path integral where the action enters via the exponent $$ \exp(iS/\hbar) $$ Note that the exponential doesn't change if we additively change $$ S \to S+2\pi \hbar = S+h$$ For this reason, the action in quantum mechanics is defined "modulo $h$": it's the fractional part of $S/h$ that may be nonzero and that is important, while the integer part is completely unphysical! You got it upside down.

Second, yes, when one is doing it right, the (theory defined by the) Nambu-Goto action may be quantized and what one gets is known as string theory (well, the first insights of string theory about a single free string, and so on). But it would be extremely awkward and ambiguous to quantize the action with the square roots etc. "directly". The clever way is to first introduce an auxiliary world sheet metric $h_{\alpha\beta}$ whose equations of motion say that it is the induced metric from the spacetime $$h_{\alpha\beta} = K\cdot \partial_\alpha X^\mu\cdot \partial_\beta X_\mu$$ where the overall normalization $K$ is deliberately left ambiguous and using $h$, the Nambu-Goto action may be rewritten as $$ S= -\frac{1}{2\pi\alpha'} \int d^2 \sigma \sqrt{-\det h}\, h^{\alpha\beta}\partial_\alpha X_\mu\partial_\beta X^\mu $$ You may check that the previous equation for $h_{\alpha\beta}$ with a certain $K$ automatically follows from this action by varying $h$. And if you substitute this value of $h$ back to the action, you get the Nambu-Goto action. So by integrating out $h_{\alpha\beta}$, you get the Nambu-Goto action back. So they're physically equivalent!

The advantage of my form, the Polyakov action, is that one may always choose world sheet coordinates so that locally $$h_{\alpha\beta}=K'\delta_{\alpha\beta}$$ and with such a simple form of the metric, the Polyakov action is just a nice free set of Klein-Gordon fields that are easy to quantize! The dependence on the scaling factor $K'$ drops between the square root of the determinant and the inverse metric in the action, a fact ("Weyl symmetry") that only holds in 2 dimensions. And the dynamics of the new Polyakov action – lots of harmonic oscillators – is still exactly equivalent physically to the original Nambu-Goto action. It seems waterproof that the nice Hilbert space one gets in this way is the only right way to "quantize the Nambu-Goto action/theory".

First, quite generally, it is not true that the action $S$ is ever required to be a multiple of Planck's constant $h$. What quantum mechanics implies is pretty much exactly the opposite thing. The formulation of quantum mechanics (any quantum mechanical theory) that uses the action is the Feynman path integral where the action enters via the exponent $$ \exp(iS/\hbar) $$ Note that the exponential doesn't change if we additively change $$ S \to S+2\pi \hbar = S+h$$ For this reason, the action in quantum mechanics is defined "modulo $h$": it's the fractional part of $S/h$ that may be nonzero and that is important, while the integer part is completely unphysical! You got it upside down. $S$ may be $3.76h$ or $3.24h$ and the difference matters; however, the difference between $S=6.64h$ and $S=8.64h$ doesn't affect (quantum) physics, a fact that's very important e.g. in Chern-Simons theories.

Second, yes, when one is doing it right, the (theory defined by the) Nambu-Goto action may be quantized and what one gets is known as string theory (well, the first insights of string theory about a single free string, and so on). But it would be extremely awkward and ambiguous to quantize the action with the square roots etc. "directly". The clever way is to first introduce an auxiliary world sheet metric $h_{\alpha\beta}$ whose equations of motion say that it is the induced metric from the spacetime $$h_{\alpha\beta} = K\cdot \partial_\alpha X^\mu\cdot \partial_\beta X_\mu$$ where the overall normalization $K$ is deliberately left ambiguous and using $h$, the Nambu-Goto action may be rewritten as $$ S= -\frac{1}{2\pi\alpha'} \int d^2 \sigma \sqrt{-\det h}\, h^{\alpha\beta}\partial_\alpha X_\mu\partial_\beta X^\mu $$ You may check that the previous equation for $h_{\alpha\beta}$ with a certain $K$ automatically follows from this action by varying $h$. And if you substitute this value of $h$ back to the action, you get the Nambu-Goto action. So by integrating out $h_{\alpha\beta}$, you get the Nambu-Goto action back. So they're physically equivalent!

The advantage of my form, the Polyakov action, is that one may always choose world sheet coordinates so that locally $$h_{\alpha\beta}=K'\delta_{\alpha\beta}$$ and with such a simple form of the metric, the Polyakov action is just a nice free set of Klein-Gordon fields that are easy to quantize! The dependence on the scaling factor $K'$ drops between the square root of the determinant and the inverse metric in the action, a fact ("Weyl symmetry") that only holds in 2 dimensions. And the dynamics of the new Polyakov action – lots of harmonic oscillators – is still exactly equivalent physically to the original Nambu-Goto action. It seems waterproof that the nice Hilbert space one gets in this way is the only right way to "quantize the Nambu-Goto action/theory".

Let me emphasize that the trick with the auxiliary metric works for the classical equations, too. The original Nambu-Goto action leads to seemingly complicated, non-polynomial differential equations of motion. But when one looks at the situation a bit carefully and uses clever methods, he finds out that the system is solvable – and it's intrinsically a set of ordinary wave equations.