Is there a physical difference between the colors of quarks? Is there any physical difference to the way, say, red quarks behave compared to green or blue ones? Or is it just an intrinsic property that they have that doesn't provide any physical difference other than that it allows two of the same to be in the same hadron?
 A: Flavors are to designate the weak interactions of the quarks.
EM charge is to designate the EM interactions of the quarks.
Colors are to designate the strong interactions. The color with the strong force is always attractive, but it can come in neutralized combinations so that you can get stable bound quarks.
Your question is whether colors are arbitrarily assigned. Yes, they are. There is no physical difference. please see this from wikipedia and another question:
What's the difference between Quark Colors and Quark Flavours?


Just as the laws of physics are independent of which directions in space are designated x, y, and z, and remain unchanged if the coordinate axes are rotated to a new orientation, the physics of quantum chromodynamics is independent of which directions in three-dimensional color space are identified as blue, red, and green. SU(3)c color transformations correspond to "rotations" in color space (which, mathematically speaking, is a complex space). Every quark flavor f, each with subtypes fB, fG, fR corresponding to the , forms a triplet: a three-componentquantum field which transforms under the fundamental representation of SU(3)c. 


