One of the main sources of subtlety in the AdS/CFT correspondence is the role played by boundary terms in the action. For example, for a scalar field in AdS there is range of masses just above the Breitenlohner-Freedman bound where there are two possible quantizations and which one you get depends on what boundary terms you add to the action. Boundary terms are also essential in the treatment of first-order Lagrangians for fermions and self-dual tensor fields. These all involve the "UV" boundary as $z \rightarrow 0$ in Poincare coordinates. Then there are dual models of QCD like the hard-wall model where one imposes an IR cutoff and imposes boundary conditions at the IR boundary and/or adds IR boundary terms to the action. My question is a bit vague, but basically I would like references to reviews, books or papers that give a good general treatment of the variational principle when one has to be careful about boundary terms. It would help if they clearly distinguish the requirements that follow from mathematical consistency from those that are imposed because of a desire to model the physics in a certain way.
I'm not sure how far back you want to go, but one of the earliest "careful" treatments (in the Hamiltonian formalism) is in the paper by Regge and Teitelboim:
Analogous work in the Lagrangian formalism didn't happen until much more recently. See the paper by Mann and Marolf, or the treatment in this relevant but shameless self-promotion:
As far as calculations specific to AAdS spacetimes there is the paper by Skenderis and Papadimitriou
which attempts to formulate things in a language similar to the Wald paper referenced by Moshe. All of these papers are concerned with the consistent treatment of the variational problem on spacetimes with a spatial infinity. The references of the last two will turn up additional relevant works.
You should immediately discount any paper which claims that the inclusion of the Gibbons-Hawking-York surface term gives a good variational problem for a gravitational theory.
This topic -- the proper variational formulation of gravitational theories -- is one of my main interests. I'm trying to take the Red Line from Loyola down to U of C for seminars this term (as teaching allows), so perhaps we can talk about it some time.
Here is a general treatment, Ishibashi and Wald, in a slightly more formal language than normal in this type of discussion. Not sure if this is what you are looking for though.