What are the boundary conditions associated to this lagrangian? Suppose that $L(q^i, \dot{q}^i)$ is a standard and well behaved lagrangian associated to some Dirichlet boundary conditions : $q^i(t_1) = q_1^i$ and $q^i(t_2) = q_2^i$.  Now I have this new lagrangian :
\begin{equation}
L' = L - \frac{d}{dt}(\lambda \: q^i \, p_i),
\end{equation}
where $\lambda$ is a positive constant.  What are the boundary conditions associated to this ?
The only thing I know is, if $\lambda = 1$, then $L'$ should be associated to fixed canonical momentum at the endpoints : $p_i(t_1) = p_{i1}$ and $p_i(t_2) = p_{i2}$ (the boundary conditions are defined by the canonical momentum).
But then what if $\lambda \ne 1$ ?
 A: I) Hamiltonian interpretation. Given a Hamiltonian $H(z;t)$ with canonical coordinates 
$$\tag{1} (z^1,\ldots,z^{2n}) ~=~ (q^1, \ldots, q^n;p_1,\ldots, p_n), $$
the Hamiltonian Lagrangian reads
$$\tag{2} L_H(z,\dot{z};t) ~=~p_k \dot{q}^k - H(z;t). $$
Then OP's modified Hamiltonian Lagrangian becomes
$$\tag{3}\tilde{L}_H(z,\dot{z};t)
~:=~L_H(z,\dot{z};t)- \lambda\frac{d}{dt}(q^k p_k)
~=~(1-\lambda)p_k \dot{q}^k -\lambda q^k\dot{p}_k- H(z;t)  .\qquad$$
Note that we can view (3) as a Lagrangian in twice as many variables (1). 
The corresponding momenta read
$$\tag{4} Q_k~:=~ \frac{\partial L_H}{\partial \dot{q}^k}~=~(1-\lambda)p_k, \qquad P^k~:=~ \frac{\partial L_H}{\partial \dot{p}_k}~=~-\lambda q^k. $$
From the general theory (as e.g. reviewed in my Phys.SE answer here), we conclude that when we vary the Hamiltonian action 
$$\tag{5}\tilde{I}_H[q]~:=~\int_{t_i}^{t_f} \! dt~ \tilde{L}_H $$ 
the boundary terms are
$$\tag{6} \left[ Q_k~\delta q^k + P^k~\delta p_k\right]_{t=t_i}^{t=t_f}
~=~\left[ (1-\lambda)p_k~\delta q^k -\lambda q^k~\delta p_k\right]_{t=t_i}^{t=t_f}.  $$
These boundary terms (6) should vanish in order for the functional/variational derivative of the Hamiltonian action (5) to exist.
Now let us discuss possible boundary conditions (BCs). If time is cyclic, so that $t_i$ and $t_f$ represent same instant, we should impose periodic BCs. (We ignore anti-periodic BCs and fermions here.) 
Let us for the remainder of this answer assume that time is not cyclic, so that initial BC and final BC are physically independent. Let us just discuss the initial BC, since the final BC is similar. We see that we need to impose at least $n$ initial BCs because there are $2n$ canonical coordinates (1).
Also it seems physically reasonable to assume that the BC treats the position coordinates on same footing. Similarly for the momentum coordinates. In other words, we are going to assume that the theory (including BCs) behaves in a natural way under
linear canonical transformation of coordinates 
$$\tag{7} q^{\prime k}~=~M^k{}_{\ell}~ q^{\ell}, \qquad p_{\ell}~=~p^{\prime}_{k} ~M^k{}_{\ell}, \qquad k,\ell~\in~\{1,\ldots, n\} . $$
(This needs not be the case, but we will assume it.) It follows that the initial BC must fix all the positions or all the momenta.
We conclude that: 


*

*For any $\lambda$, one can choose initial BC $q(t_i)=0$ or $p(t_i)=0$.

*For $\lambda=0$, one can also choose initial BC $q(t_i)=q_i$. 

*For $\lambda=1$, one can also choose initial BC $p(t_i)=p_i$. 
II) Lagrangian interpretation. Given a first-order Lagrangian $L(q,\dot{q};t)$, the Lagrangian momenta reads
$$\tag{8} p_k(q,\dot{q};t)~:=~\frac{\partial L(q,\dot{q};t)}{\partial \dot{q}^k}$$
Then OP's modified Lagrangian 
$$\tag{9} \tilde{L}(q,\dot{q},\ddot{q};t) 
~:=~ L(q,\dot{q};t) - \lambda\frac{d}{dt}\left(q^kp_k(q,\dot{q};t)\right)$$
becomes of second-order for $\lambda\neq 0$. Note that for a higher-order theory, one should in general impose more boundary conditions, cf. e.g. my Phys.SE answer here.
The EL eqs. 
$$\tag{10} \frac{\partial \tilde{L}}{\partial q^k} -\frac{d}{dt}\frac{\partial \tilde{L}}{\partial \dot{q}^k}+\left(\frac{d}{dt}\right)^2 \frac{\partial \tilde{L}}{\partial \ddot{q}^k} - \ldots~\approx~0 $$
for the Lagrangian (9) are not changed by a total derivative term, and are still generically of second order.
However, for $\lambda\neq 0$, we have to impose four BCs. Since the EL eqs. are of second order, there will generically not be any stationary solutions. We conclude that OP's variational problem is ill-defined for $\lambda\neq 0$.
A: The following is rewritten after @Qmechanic's comment. While his observation was correct, I think the main point below holds on its own. 
The case @Cham considers is that of a Lagrangian $L' = L - \frac{d}{dt}q^ip_i$ modified by a total derivative for the purpose of implementing a change in the boundary conditions. Originally the $p_i$-s are assumed to be the canonical momenta $p_i = \frac{\partial L}{\partial \dot{q}^i}$, which makes the total derivative a function of the $q^i$, $\dot{q}^i$. We can relax the assumption on the particular form of the total derivative and generalize instead to 
$$
L'(t, q, \dot{q}) = L(t, q, \dot{q}) - \frac{d}{dt}F(t, q, \dot{q})
$$ 
Here is the problem: The total derivative leaves the EL eqs unchanged, but using natural boundary conditions leads to an implicit condition for $F$:
$$
\frac{\partial F}{\partial \dot{q}^i} = 0
$$
The modified Lagrangian must necessarily read
$$
L'(t, q, \dot{q}) = L(t, q, \dot{q}) - \frac{d}{dt}F(t, q)
$$
In other words, the original choice $F = \lambda q^ip_i(t, q, \dot{q})$ is ill-defined for this purpose: taking the $p_i$ to mean canonical momenta, or even any function such that $\partial p_i / \partial \dot{q}^i \neq 0$, leaves the boundary conditions unresolved. Hence the $p_i$-s must be functions of time and/or the $q^i$-s only, $p_i = p_i(t, q)$. 

Why this is so:
The contribution of any $\frac{dF}{dt} = \frac{d}{dt}F(t, q, \dot{q})$ to the variation of the action is confined to boundary terms:
$$
\delta I = \int_{t_1}^{t_2}{dt\;\delta\left( L(t, q, \dot{q}) - \frac{dF}{dt}(t, q, \dot{q})\right)} = \int_{t_1}^{t_2}{dt\;\delta L(t, q, \dot{q})} - \delta F(t, q, \dot{q})\Big|_{t_1}^{t_2} =\\
\int_{t_1}^{t_2}{dt\;\left( \frac{\partial L}{\partial q^i}  - \frac{d}{dt}\frac{\partial L}{\partial \dot{q}^i}\right)\delta q^i} + \left[\left(\frac{\partial L}{\partial \dot{q}^i} - \frac{\partial F}{\partial q^i}  \right)\delta q^i - \frac{\partial F}{\partial \dot{q}^i}\delta \dot{q}^i\right]\Big|_{t_1}^{t_2}
$$
We may try to find some $G = G(t, q, \dot{q})$ such that 
$$
\left(\frac{\partial L}{\partial \dot{q}^i} - \frac{\partial F}{\partial q^i}  \right)\delta q^i - \frac{\partial F}{\partial \dot{q}^i}\delta \dot{q}^i = \delta G
$$
in which case the boundary conditions would amount to $\delta G\Big|_{t=t_{1,2}} = 0$. But this would require
$$
\frac{\partial L}{\partial \dot{q}^i} - \frac{\partial F}{\partial q^i} = \frac{\partial G}{\partial q^i}\\
- \frac{\partial F}{\partial \dot{q}^i} = \frac{\partial G}{\partial \dot{q}^i}
$$ 
which, assuming $\frac{\partial^2 G}{\partial \dot{q}^i \partial q^j} = \frac{\partial^2 G}{\partial q^j \partial \dot{q}^i }$, would lead to
$$
\frac{\partial^2 G}{\partial \dot{q}^i \partial q^j} = \frac{\partial^2 L}{\partial \dot{q}^i \partial \dot{q}^j} - \frac{\partial^2 F}{\partial \dot{q}^i \partial q^j} = - \frac{\partial^2 F}{\partial \dot{q}^i \partial q^j}
$$
and 
$$
\frac{\partial^2 L}{\partial \dot{q}^i \partial \dot{q}^j} = 0
$$
This follows even if we try $F = \lambda q^i\frac{\partial L}{\partial \dot{q}^i} = \lambda q^i p_i$. So unless $L$ is linear in the $\dot{q}^i$, the two boundary contributions must vanish simultaneously for any $\delta q$, $\delta \dot{q}$, although only one of them can give boundary conditions for a first order Lagrangian. The other must vanish identically for any $t$, $q$, $\dot{q}$. Requesting this of the first term doesn't work since the 2nd one would impose additional conditions on the Lagrangian. This leaves
$$
\frac{\partial F}{\partial \dot{q}^i} = 0
$$
and boundary conditions of the form
$$
\left(\frac{\partial L}{\partial \dot{q}^i} - \frac{\partial F}{\partial q^i}  \right)\Big|_{t=t_{1,2}} = 0
$$
Incidentally the requirement that $\frac{\partial F}{\partial \dot{q}^i} = 0$ removes the issue that the modified Lagrangian $L' = L - \frac{dF}{dt}$ could become second order. 
Note however that assuming $F=F(t, q, \dot{q})$ does not imply by itself that the EL equations for $L'$ are necessarily higher order. This is because although $\frac{dF}{dt}$ is linear in the $\ddot{q}^i$, when taking the variation under the action integral the terms in $\delta \ddot{q}^i$ get integrated by parts twice and cancel the remaining contributions from $F$, leaving only the EL terms in $L$ and the corresponding boundary terms.
A: I'm posting here an extraction from a paper written by T. Padmanabhan (http://arxiv.org/abs/hep-th/0608120) :
Consider  a dynamical variable  $q(t)$ in point mechanics described by a lagrangian $L_q(q,\dot q)$.  Varying the action obtained from integrating this Lagrangian in the interval $(t_1,t_2)$ and keeping $q$ fixed at the endpoints, gives the Euler Lagrange equations for the system $(\partial L_q/\partial q)=dp/dt$, where we have defined a function $p(q,\dot q)\equiv (\partial L_q/\partial \dot q)$. (The subscript $q$ on $L_q$ is an indicator of the variable that is kept fixed at the end points.) The lagrangian contains only up to first derivatives of the dynamical variable and the equations of motion are - in general - second degree in the time derivative. 
When the lagrangian $L_q$  depends on $\ddot q$ as well, the theoretical formulation becomes more complicated. For example, if the equations of motion become higher order, then more initial conditions are required to pose a well-defined initial value problem and the corresponding definition of path integral in quantum theory, using the lagrangian, is nontrivial.  Interestingly enough, there exists a wide class of lagrangians $L(\ddot q,\dot q,q)$ which depend on $\ddot q$ but still lead to equations of motion which are only second order in time.
Let consider the following question: We want to modify the lagrangian $L_q$ such that  the same equations of motion are obtained when  - instead of fixing $q$ at the end points - we  fix some other (given) function $C(q,\dot q)$  at the end points. This is easily achieved by modifying the lagrangian by adding a term $-df(q,\dot q)/dt$ which depends on $\dot q$ as well. (The minus sign is just for future convenience.) 
The new lagrangian is:
\begin{equation}\tag{1}
L_C(q,\dot q,\ddot q)=L_q(q,\dot q)-\frac{df(q,\dot q)}{dt}
\end{equation}
We want this lagrangian $L_C$ to lead to the same equations of motion as $L_q$, when some given function $C(q,\dot q)$  is held fixed at the end points. We assume $L_q$ and $C$ are given and we need to find $f$. The standard variation gives
 \begin{equation}\tag{2}
\delta A_C =\int_{t_1}^{t_2} dt\left[\left(\frac{\partial L}{\partial q}\right)-\frac{dp}{dt}\right]\delta q
-\int_{t_1}^{t_2} dt \frac{d}{dt}\left[\delta f -p \; \delta q \right]
\label{elf}
\end{equation} 
 We  will now invert the relation $C=C(q,\dot q)$ to determine $\dot q=\dot q(q,C)$ and  express $p(q,\dot q)$ in terms of $(q,C)$ obtaining the function $p=p(q,C)$.  In the boundary term in (2) we treat $f$ as a function of $q$ and $C$, so that the variation of the action can be expressed as:
\begin{align}
\delta A_C=&\int_{t_1}^{t_2} dt\left[\left(\frac{\partial L}{\partial q}\right)-\frac{dp}{dt}\right]\delta q+
\left[p(q,C)-\left(\frac{\partial f}{\partial q}\right)_C\right]\delta q\Big|_{t_1}^{t_2}  -\left(\frac{\partial f}{\partial C}\right)_q\delta C\Big|_{t_1}^{t_2} \\
=&\int_{t_1}^{t_2} dt\left[\left(\frac{\partial L}{\partial q}\right)-\frac{dp}{dt}\right]\delta q+
\left[p(q,C)-\left(\frac{\partial f}{\partial q}\right)_C\right]\delta q\Big|_{t_1}^{t_2} \tag{3}
\end{align}
since $\delta C=0$ at the end points by assumption. To obtain the same Euler-Lagrange equations,
the second term should vanish for any $\delta q$. This fixes the form of $f$ to be:
\begin{equation}\tag{4}
f(q,C)=\int p(q,C)dq +F(C)
\label{f}
\end{equation}
where the integration is with constant $C$ and $F$ is an arbitrary function. 
Thus, given a lagrangian $L_q(q,\dot q)$ which leads to certain equations of motion when $q$ is held fixed, one can construct a family of lagrangians $L_C(q,\dot q,\ddot q)$ which will lead to the {\it same} equations of motion when an arbitrary function $C(q,\dot q)$ is held fixed at the end points. This family is remarkable in the sense that $L_C$ will be a function of not only $q,\dot q$, but will also involve $\ddot q$. In spite of the existence of the $\ddot q$ in the lagrangian, the equations of motion are still of second order in $q$ because of the special structure of the lagrangian. (The results obtained above have an interpretation in terms of canonical transformations etc. which we purposely avoid since we want to stay within the lagrangian framework). So, even though a general lagrangian which depends on $\ddot q$ will lead to equations of higher order, there is a host of lagrangians with a special structure which will not.
A: Lets define the following action :
    \begin{equation}\tag{1}
  S' = \int_{t_1}^{t_2} L' \, dt,
 \end{equation}
where
    \begin{equation}\tag{2}
  L' = L - \lambda \, \frac{d}{dt} (q^k \, p_k).
 \end{equation}
An arbitrary variation $\delta q^k$ gives the following :
\begin{align}
\delta S' &= \int_{t_1}^{t_2} \Big( \frac{\partial L}{\partial q^i} \; \delta q^i + \frac{\partial L}{\partial \dot{q}^i} \; \delta \dot{q}^i \Big) \, dt - \lambda \, \delta (q^k \, p_k) \Big|_{t_1}^{t_2} \\[12pt]
&= \int_{t_1}^{t_2} \Big[ \frac{\partial L}{\partial q^i} - \frac{d}{dt} \Big( \frac{\partial L}{\partial \dot{q}^i} \Big) \Big] \, \delta q^i \; dt + \int_{t_1}^{t_2} \frac{d}{dt} \Big( \frac{\partial L}{\partial \dot{q}^i} \; \delta q^i \Big) \, dt - \lambda \, \delta (q^k \, p_k) \Big|_{t_1}^{t_2} \\[12pt]
&= (\text{usual Euler-Lagrange variation of } S) + \big(p_i \; \delta q^i - \lambda \, \delta (q^k \; p_k)\big)\Big|_{t_1}^{t_2} \\[12pt]
&= (\ldots) + \big( (1 - \lambda) \, p_k \; \delta q^k - \lambda \, q^k \, \delta p_k \big) \Big|_{t_1}^{t_2}. \tag{3}
\end{align}
The first term cancels from the Euler-Lagrange equations.  To find a proper quantity that is fixed at the endpoints $t_1$ and $t_2$, the last term need to be written like this (look at the position of the indices.  The bar is here on purpose, since there are two solutions) :
    \begin{equation}\tag{4}
  (1 - \lambda) \, p_k \; \delta q^k - \lambda \, q^k \, \delta p_k = -\; Q^k \; \delta P_k \quad \text{or} \quad \bar{P}_k \; \delta \bar{Q}^k.
 \end{equation}
We could then fix $P_k$ or $\bar{Q}^k$ at $t_1$ and $t_2$ : $\delta P_k(t_1) = 0$ and so on.  The two solutions are apparently a consequence of the symmetry between the variables $q^k$ and $p_k$ in the product $q^k \, p_k$.  From the indices position, we can guess the following :
    \begin{align}
  Q^k &= \mathcal{Q}(q, p) \, q^k,
  & P_k &= \mathcal{P}(q, p) \, p_k, \tag{5} \\[18pt]
  \bar{Q}^k &= \bar{\mathcal{Q}}(q, p) \, q^k,
  & \bar{P}_k &= \bar{\mathcal{P}}(q, p) \, p_k. \tag{6}
 \end{align}
From dimensional ground and indices position, we could expect that $\mathcal{P}(q, p) \propto (q^i \, p_i)^a$ and $\bar{\mathcal{Q}}(q, p) \propto (q^i \, p_i)^b$, where $a$ and $b$ are unknown exponents to be found as functions of $\lambda$.  The variation of $P_k$ and $\bar{Q}^k$, substituted into the equation above, give us some simple algebraic equations that are easy to solve :
    \begin{equation}
  -\; a \, \mathcal{Q} \, (q^i \, p_i)^a \, p_k \; \delta q^k - (1 + a) \, \mathcal{Q} \, (q^i \, p_i)^a \, q^k \, \delta p_k = (1 - \lambda) \, p_k \; \delta q^k - \lambda \, q^k \, \delta p_k,
 \end{equation}
or :
    \begin{equation}
  (1 + b) \, \bar{\mathcal{P}} \, (q^i \, p_i)^a \, p_k \; \delta q^k + b \, \bar{\mathcal{P}} \, (q^i \, p_i)^a \, q^k \, \delta p_k = (1 - \lambda) \, p_k \; \delta q^k - \lambda \, q^k \, \delta p_k.
 \end{equation}
The complete algebraic solutions are then $a = \lambda - 1$, $b = - \lambda$ and
    \begin{align}
  \mathcal{Q}(q, p) &= (q^i \, p_i)^{1 - \lambda},
  & \mathcal{P}(q, p) &= (q^i \, p_i)^{\lambda - 1}, \\[18pt]
  \bar{\mathcal{Q}}(q, p) &= (q^i \, p_i)^{- \lambda},
  & \bar{\mathcal{P}}(q, p) &= (q^i \, p_i)^{\lambda}.
 \end{align}
Finally, the quantities that are fixed at the endpoints are these :
    \begin{equation}\tag{7}
  P_i = (q^k \, p_k)^{\lambda - 1} \, p_i,
 \end{equation}
or :
    \begin{equation}\tag{8}
  \bar{Q}^i = (q^k \, p_k)^{- \lambda} \, q^i.
 \end{equation}
Now I'm still puzzled as why there are two solutions.  Also, since $q^i \, p_i$ may turn out negative, these quantities may be ill-defined for some values of $\lambda$ ($\lambda = \tfrac{1}{2}$, for example).
Some special cases :
If $\lambda = 0$, the natural solution is given by the second one :  $\bar{Q}^i = q^i$ (Dirichlet conditions).  Why is there another solution (first one above) :
    \begin{equation}\tag{9}
  P_i = \frac{p_i}{q^k \, p_k} \; ?
 \end{equation}
If $\lambda = 1$, then the natural solution is given by the first one : $P_i = p_i$.  Why is there another solution (second one above) :
    \begin{equation}\tag{10}
  \bar{Q}^i = \frac{q^i}{q^k \, p_k} \; ?
 \end{equation}
Someone confirms the calculations above, and have an interpretation of the dual solutions ?  And what about the sign problem, for some specific values of $\lambda$ ?
EDIT 1 : It is funny to notice that the $Q^i$ and $P_i$ (or $\bar{Q}^i$ and $\bar{P}_i$) defined above (expressions (5) or (6)) are canonical transformations of the $q^i$ and $p_i$ :
\begin{align}\tag{11}
\{ Q^i, \, Q^j \} &= 0,
&\{ P_i, \, P_j \} &= 0,
&\{ Q^i, \, P_j \} &= \delta^i_j.
\end{align}
EDIT 2 :  The expression $q^k \, p_k$ is invariant under the canonical transformation
\begin{align}\tag{12}
Q^k &= (q^i p_i)^{-1} \, q^k,
& P_k &= (q^i p_i) \, p_k, \\[12pt]
q^k &= (Q^i P_i) \, Q^k,
& p_k &= (Q^i P_i)^{-1} \, P_k, \tag{13}
\end{align}
so $q^k \, p_k \equiv Q^k \, P_k$, and it's easy to verify the identity
\begin{equation}\tag{14}
- \; q^k \; \delta p_k \equiv P_k \; \delta Q^k.
\end{equation}
So it is not surprising after all that there's a second solution to the problem.  This $Q^k$ may be interpreted as another quantity that is fixed at the boundaries, in the case of $\lambda = 1$ ($p_k$ and $Q^k$ are both "natural" quantities that are fixed in this case).
