The ability to deliver energy relatively quickly is basically the distinction between a "capacitor" and a "rechargeable battery". This isn't a physics factoid so much as just what the words mean.
For example, in the below plot:
$
{\require{color}}
{\definecolor{capacitor}{RGB}{255,10,10}}
{\definecolor{lightCapacitor}{RGB}{255,131,131}}
{\definecolor{battery}{RGB}{186,138,20}}
{\definecolor{lightBattery}{RGB}{219,194,133}}
%
%\text{For example, in the below plot:} \\
\hskip{1em}
\lower{2.5ex}{
\begin{array}{l}
{\rlap{\color{capacitor}{\rule{15px}{15px}}}}
{\rlap{\raise{4px}{\hskip{4px} \color{lightCapacitor}{\rule{7px}{7px}}}}}
\hskip{21px}
{\raise{2px}{
{\color{capacitor}{\textbf{Li-ion capacitor}}}
~\text{has a higher discharge rate; though}
}}
\\
{\rlap{\color{battery}{\rule{15px}{15px}}}}
{\rlap{\raise{4px}{\hskip{4px} \color{lightBattery}{\rule{7px}{7px}}}}}
\hskip{21px}
{\raise{2px}{{\color{battery}{\textbf{Li-ion battery}}}
~\text{can store more energy.}}}
\end{array}
}
$
$\hskip{50px}$$
{\require{cancel}}
{\def\place#1#2#3{\smash{\rlap{\hskip{#1px}\raise{#2px}{#3}}}}}
\place{305}{219}{\color{capacitor}{\bcancel{\phantom{\rule{97px}{25px}}}}}
\place{377}{191}{\color{battery}{\cancel{\phantom{\rule{25px}{7px}}}}}
$
Note that power has units of $\left[\frac{\text{energy}}{\text{time}}\right]$. This is, power is the rate at which energy's delivered.
Conceptually, there seems to be a conflict-of-interest between storing energy and being able to rapidly lose it (i.e., deliver power). As shown above, particular technologies tend to have a trade-off between their ability to store and deliver energy.
This conflict may be seen as similar to that with thermodynamic reversibility in which slower processes tend to have higher efficiencies. For example, useful heating has the highest thermodynamic efficiencies when it flows down arbitrarily small temperature gradients, though the smaller the temperature gradient, the longer it takes for a finite amount of heat to move across it.
In thermodynamics, a reversible process is a process whose direction can be "reversed" by inducing infinitesimal changes to some property of the system via its surroundings, with no increase in entropy. Throughout the entire reversible process, the system is in thermodynamic equilibrium with its surroundings. Since it would take an infinite amount of time for the reversible process to finish, perfectly reversible processes are impossible.
–"Reversible process (thermodynamics)", Wikipedia [formatting and references omitted]
It's actually kinda fun to think about the information-theoretic aspects about why this is. For example, you've probably heard about how entropy is a measure of disorder; it's perhaps more properly seen as a qualification of how states in an ensemble of possible states could flow. When there're more unbound flow pathways, things can move faster; however, that also means that entropy grows, leaking useful work.
Also, that leaking of useful work comes out as thermal energy (heat), which can be pretty problematic when it comes to high-voltage electronics.
As a historical note, capacitors used to be more physical mechanisms for storing energy while batteries used to be more chemical mechanisms for storing energy (with some funny exceptions). This continues to often be true today, though that's perhaps better seen as historical happenstance than as a basic concept to keep track of. Stuff like supercapacitors and other technologies'll continue to blur the line, since there's really no reason for a well-engineered system to be limited to a single physical approach.
As a final note, a defibrillators could use batteries for their principal energy storage, using them to charge capacitors that could rapidly discharge. This design pattern's called transient load decoupling, where the transient load is the electrical demand of the shock and the decoupling is how the battery has less direct exposure to it.