The simple answer would be: consider a physical system that can be in two states. Its thermodynamic entropy is $k_B \log(2)$ due to this. When you erase the bit, then you have a definite microstate and the entropy is zero. Since entropy cannot decrease it has to go somewhere, and this means it needs to be transferred to the environment that has temperature $T$, giving us a cost $k_B T \log(2)$.
This is the simplified version of a slightly dodgy derivation (note how I facilely equated thermodynamic entropy with information entropy without justification and did not specify how erasure happened or the entropy got moved around). Once can do this more rigorously, like in this paper.
Also, it is worth pointing out it is a misconception that the cost has to be paid in energy. If you have an empty computer memory you can just swap the erased bit for a fresh zero reversibly. But empty computer memory is in a sense a heat reservoir at absolute zero but with very limited capacity (it will eventually run out). It is possible to show that the cost can be paid using other conserved quantities like spin. It is just that in practice we tend to move entropy around by dumping it as waste heat in a cool outside heat bath.