Coefficient of friction is between two materials. So rubber would have a different coefficient of friction on pavement, than on glass. That said, some materials, like rubber, tend to have higher coefficients of friction with many other materials. Other materials, like PTFE and ice, have low coefficients of friction with many other materials. Ice is often given as an example of a material that has a low coefficient of friction. Below is a summary explanation of some of the effects that influence coefficient of friction. For more detail, this topic falls under the category of tribology.
Surface Roughness +
If two materials are both extremely rough (like <200 grit sandpaper) then the parts that stick up on one surface (asperities) will often find their way into the valleys of the other surface, like gear teeth. This interaction helps prevent the surfaces from sliding against each other. Thus, higher roughness materials will often have higher coefficients of friction.
Often times there will be some media that is easily deformable that sits between the two surfaces. The oil in your engine separates the steel parts so well that the steel doesn't actually touch at all. In this case the friction force is actually due to the viscous effects of the oil deforming, and is highly velocity dependent. A wet road has a lower coefficient of friction with rubber than a dry road, due to the water acting as a lubricant. Ice will self lubricate by melting, providing a layer of water to separate the ice crystals from the other surface. Other solids, like graphite, will self lubricate by breaking off tiny bits of themselves that will then act as lubrication.
Intermolecular forces +
When molecules get very close together they tend to interact. Often they want to hold onto each other, through the Van der Waals force or otherwise. This tends to increase friction. PTFE (trandnamed Teflon) tends to have very low intermolecular forces and as a result it tends to have low friction.
There are a lot of "tends to" and "often"s in the above paragraphs due to how these (and other) effects interact. For example, increasing surface roughness can actually decrease friction if it allows for lubrication to better fill the gaps between the surfaces. It could also decrease friction by just separating most of the molecules from the other surface so the intermolecular forces are less. The stiffness of surfaces comes into play as one surface deforms another. For example rubber deforms very easily, which allows it to match the surface roughness of the other material, interlocking into it, and reducing the intermolecular distances to increase the intermolecular forces. This is one reason rubber tends to have high friction. Brass on the hand, while it also deforms easily, tends to deform in a way that rips off asperities, that then form lubricating particles. So brass's softness actually decreases friction.