I have written similar things in other answers. But getting this message out depends on repeating it to some extent.
John Yeager, a well-respected rock mechanics expert, has a famous quote that goes something like this: "There are two things you need to know about [the coefficient of] friction. It is always 0.6 and it will always make a monkey out of you."
There are (at least) three different levels of "laws" that we learn in physics and engineering. The first level of laws are things like $\Sigma \textbf{F} = m \textbf{a}$ and its angular counterpart. These laws are extremely accurate (say, maybe one part in a million) for everyday objects that move much slower than the speed of light.
The next level of laws are called "constitutive" that describe material behavior. Laws of this sort would include $\bf \sigma = E \epsilon$. They are also quite accurate, on the order of maybe 1% or better under controlled experimental conditions.
The last and least accurate set of physical laws, including friction effect $F = \mu N $, coefficient of restitution, etc. are rough approximations. Calling them "laws" is probably a misnomer. Although they are convenient for teaching concepts, they can give you a +/- 20% estimate at best, but should not be treated as "laws" in the same sense of constitutive laws or Newton-Euler laws. Slightly different conditions will lead to different measurements and answers, as any tribologist (friction scientist) will tell you. It is an unfortunate feature of many physics and engineering courses that they fail to point out the fundamental differences in the accuracy of these laws. So students are left thinking that they are all as valid as the other.
So, the answer to your question is, basically, "it's complicated." I suspect that there is some effect from contact PRESSURE which takes the surface area into account. But that is just a guess. One way, maybe, to get an improvement in your results would be to use more slippery surfaces. The weirdness gets a bit better when things are smooth and/or oiled.
Credit for these observations goes to my teacher and advisor Andy Ruina from Cornell University. If you want to read his version of the above and extended commentary, I recommend his book "Introduction to Statics and Dynamics," co-authored with Rudra Pratap.