1) If we take the heat out of the system (i.e. with a cooling circuit, like happens in car engines) would gas temperature go down to the initial state, without decompressing it?
Is the piston fixed? Then the volume will remain constant during the cooling. Gas temperature would go down and pressure will fall.
How far down the temperature will go depends on the cooling. If the cooling circuit is bound to a lower temperature, then it will continue cooling until the gas has reached that. Independent of any initial temperature of the gas.
2) How long could you repeat that compression-heating-cooling process? Would it result in liquefying the gas?
If I understand it correctly, you remove the heat caused by compression (and return to initial temperature), then compress it again, then remove the heat and repeats like that.
This is exactly what happens in a usual pressure cylinder - like and air cylinder for diving. That will have a large pressure inside because a lot of air is compressed, but the temperature is equal to the surroundings.
Yes, it will liquify at some point. You compress it, pressure rises and the boiling point also rises. It will stay as gas, if the temperature is allowed to rise also. But by removing heat and cooling it down, you will at some point have a temperature below the new boiling point. In a phase diagram of a substance you will be able to see where the transition points are from liquid to gas - that point is tightly bound to volume, pressure and temperature.
3) How can I calculate the relation between the piston speed, pressure, and the heat generated?
Why piston speed? If you move the piston fast, you add kinetic energy to the molecules not caused by the volume decrease. You will have a hard time figuring out how much is due to piston movement.
Usually in models of this kind of volume decrease in a pressure champer the piston is assumed to move so slowly that kinetic energy is negligible. Then piston speed will have no relation to the other parameters.