I keep encountering the phrase a bit of information but I don't know precisely what this means. I have an intuition of a system that can be in 1 of 2 possible states (in the classical realm that is) which we can label $|1\rangle$ or $|0\rangle$. One might say that specifying what state the system is in at one particular instant conveys a bit of information. But I'm not sure if this is a proper way of thinking about it so what exactly is the formal definition of a bit in physics?
Bits are not defined in physics. They are defined in information theory, and then physics uses them in cases where their physical attributes they care about are well modeled using information theory.
The most common case we see this is exactly the case you mention: where we know the system is in some subset of a countable number of states. In these cases, we can talk about the information we know about a system in terms of bits. In equilibrium thermodynamics, this is often a very convenient way of thinking about things.
This leads to Laudauer's Principle, which is probably the closest to a "definition of a bit' physics will have. Landauer's principle states that the minimum amount of energy it takes to erase one bit of information is $kT\ln 2$, where k is Boltzmann's constant and T is the temperature of the system. However that "definition" only applies in a few key situations (equilibrium thermodynamics) and even in that environment, the assumptions that he made to derive that equation are questioned from time to time. We have seen experimental evidence to suggest that this principle does not hold in non-equilibrium thermodynamic scenarios, so you can't call it a "definition of a bit in Physics" because there's more to it.