How does a decelerating universe agree with this version of the Friedmann-Robertson-Walker formula? I was watching a video discussing dark matter, and he presented the following simplified version of the Friedmann-Robertson-Walker equation:
$$ \left(\frac{\dot{a}(t)}{a(t)}\right)^2 = \frac{8\pi G}{3} \rho(t) -\frac{k}{a(t)^2}$$
where $a$ is the scale factor, $\rho$ is the density, $k$ is a constant, and $G$ is the gravitational constant.
He said that if the $k$ factor is more negative than the $G$ factor, the universe expansion would have negative deceleration. However, the L.H.S term is squared so it can never be negative. So, what did he mean by negative?
 A: There are two Friedmann equations, and they are usually written as:
$$ \frac{\dot{a}^2 + kc^2}{a^2} = \frac{8\pi G \rho + \Lambda c^2}{3} \tag{1}$$
$$ \frac{\ddot{a}}{a} = \frac{-4\pi G}{3}\left(\rho + \frac{3p}{c^2}\right) + \frac{\Lambda c^2}{3} \tag{2} $$
The equation you cite is obtained from equation (1) by assuming the cosmological constant, $\Lambda$, is zero. With this assumption the equation simplifies to:
$$ \frac{\dot{a}^2 + kc^2}{a^2} = \frac{8\pi G \rho}{3} \tag{1a}$$
and subtracting $kc^2/a^2$ from both sides gives your equation (I assume that in your equation the $c^2$ has been subsumed into the constant $k$).
$$ \left(\frac{\dot{a}}{a}\right)^2 = \frac{8\pi G \rho}{3} - \frac{kc^2}{a^2} \tag{1b}$$
But then you say:

He said that if the $k$ factor is more negative than the $G$ factor, the universe expansion would have negative deceleration (my emphasis)

But equation (1b) gives us the velocity $\dot{a}$ not the acceleration (or deceleration) $\ddot{a}$. The value of $\dot{a}$ can be positive or negative because when we take the square root of $(\dot{a}/a)^2$ we get both positive or negative roots, but the right side cannot be negative or the square root would be imaginary.
It's the second equation that gives us the acceleration, but the acceleration is independent of $k$ and indeed if we assume $\Lambda = 0$ the acceleration is always negative anyway.
You have not linked the video so I cannot watch it myself to check, but either you have misunderstood what the video is claiming or the video is simply wrong.
A: The video is "Dark Energy" by DrPhysicsA, and the claim in question is at 19:45.
It's a mistake. It would have been correct to say that if $ρ(t)>0$ at all times and $k\le 0$, then the RHS is positive at all times, so there is no time at which $a'(t)=0$, so $a'(t)$ can't switch sign, so if the universe is expanding at any time then it's expanding at all times. But he definitely says that the RHS being negative means the universe is contracting, which is wrong.
It isn't the only mistake in the video. At 21:30 he says that in the critical case, the scale factor asymptotically approaches a maximum value. That's quite wrong and I don't see how it can be explained away as mere blackboard nervousness. If it were true, Einstein wouldn't have needed the cosmological constant at all: he could have just said that we are in a critical-density universe at a very late time.
At 38:30, and again at 1:04:10, he draws a sudden transition from radiation to matter dominance with a kink in the graph of $a(t)$ as though $a''(t)\gg 0$ briefly at that time. In a correct graph, there are no sudden changes in $a''(t)$ and it's negative at all times prior to $Λ$ dominance (much later).
At the end of the video he repeats the common misconception that light from galaxies with a recession velocity greater than $c$ won't reach us. He only brings up this idea when he starts talking about dark energy; he doesn't seem to realize that even in the $Λ=0, k=0$ models that he discussed earlier, galaxies more distant than $c/H$ have a recession velocity larger than $c$, and yet the light from them reaches us (there is no cosmological horizon in those models).
