Application of Lagrange Multipliers in action principle

In Goldstein's Classical Mechanics, he suggests the use of Lagrange Multipliers to introduce certain types of non-holonomic and holonomic contraints into our action. The method he suggests is to define a modified Lagrangian $$L^{'}(\dot{q},q;t) = L(\dot{q},q;t) + \sum^{m}_{i = 1}\lambda_{i} f_{i}(\dot{q},q;t),$$ where $$f_{i}(\dot{q},q;t)$$ are $$m$$ equations of constraint, and $$L$$ the original Lagrangian. He then proceeds to define the action $$S^{'} = \int_{t_{1}}^{t_{2}}L^{'}\,dt$$ and takes the variation of $$S^{'}$$ to be zero, thus applying Hamilton's principle.

My confusion in this approach arises from the way in which the Lagrange Multipliers are introduced. I don't see why $$\sum^{m}_{i = 1}\lambda_{i} f_{i}(\dot{q},q;t)$$ should be introduced inside the integral.

In multivariable calculus, the Lagrange multiplier system stems from the idea that if we want to extremize a function subject to certain constraints, then the gradient of the function will be proportional to a linear combination of the gradient of the constraint equations. Here, the function in question is the action, not the Lagrangian. So, I feel like the resolution should be that $$\delta S + \delta \sum^{m}_{i = 1}\lambda_{i} f_{i}(\dot{q},q;t) = 0;\, S = \int_{t_{1}}^{t_{2}}L\,dt$$ and not $$\delta S^{'} = 0; \, S = \int_{t_{1}}^{t_{2}}L^{'}\,dt.$$

To me, it isn't clear if this makes sense or if the two methods are equivalent.

1. It should be stressed that the constraints $$f_{\ell}(q,\dot{q},t), \qquad \ell~\in~\{1,\ldots, m\}$$ depends implicitly (and possible explicitly) of time $$t$$, so we have continuously many constraints, namely for each instant of time $$t$$.

Therefore we should introduce continuously many Lagrange multipliers $$\lambda^{\ell}(t)$$.

And therefore we should sum $$\sum_{\ell=1}^m$$ and time-integrate $$\int\! dt$$ the term $$\lambda^{\ell}(t)f_{\ell}(q,\dot{q},t)$$ in the extended action. This fact seems to answer OP's main question.

2. Finally, it should be stressed that Goldstein's treatment of non-holonomic constraints for an action principle is flawed/inconsistent, cf. e.g. this & this Phys.SE posts.

Properly speaking we should therefore assume that the constraints $$f_{\ell}(q,\dot{q},t)$$ does not depend on the generalized velocities $$\dot{q}$$, i.e. that they are holonomic $$f_{\ell}(q,t)$$.