Here's a more-or-less rigorous discussion. The work done by a
force acting over
some small distance is $dW = \vec F\cdot d\vec s$. If the work is
being done by gas with pressure $p$ pushing on some area $\vec A$
(where the "direction" of the area is perpendicular to the surface),
and moving this surface by some distance $d\vec x$, then the work over
the entire area is \begin{align} dW &= \vec F \cdot d\vec s \\ &=
(p\vec A) \cdot d\vec x \\ &= p (\vec A \cdot d\vec x) = p\,dV.
\end{align} The work per unit area is
$$
\frac{dW}A = p
\frac{dV}{A} = p\,dx,
$$
where $dx$ is again the displacement of the
surface. Then the power per unit area is
$$
\frac{dP}{A} =
\frac{dW}{A\,dt} = p\frac{dx}{dt} = pv,
$$
where $v=dx/dt$ is the
speed of the surface.
It doesn't feel like too much of a stretch to say that the direction
of the energy transfer will be in the direction of the velocity of the
surface, as well.
To see that this treatment works for sound waves, imagine
one-dimensional sound waves moving left-to-right in a pipe. (Image
source)
Make a small volume with constant mass with the location and
approximate width of one of the red arrows: as the density near the
arrow changes, your constant-mass volume will have to get bigger or
smaller to accomodate. A compression entering your small volume from
the left compresses your small volume, doing work on it. When the
compression leaves your small volume to the right, then your small
volume is expanding and doing work on the next little bit to the
right. And so in this way work gets done by each little bit of gas in
the pipe on the little bit to its right, and energy is transfered down
the pipe.
If we model the pressure as a base pressure plus an oscillating term,
$$
p = p_\text{atm} + p_\text{wave}\sin(kx-\omega t),
$$
and the particle velocity as
$$
\vec v = \vec v_\text{max} \sin(kx - \omega t),
$$
then you can see that the instantaneous power at any given position
$x$ may be positive or negative: energy enters and leaves each part of
the air column. However only the oscillating part of
the pressure, $p_\text{wave}$, will contribute to the power when
averaged over many periods. The average power in the tube
will therefore go like $P = p_\text{wave} v_\text{max} A$, in the same
direction as $\vec v$.