Mass is one of the most core and complicated concepts in dynamics.
It isn't complicated. There's some ambiguities, in that there's inertial mass and active gravitational mass and relativistic mass and more. But nowadays when we say mass without qualification we mean rest mass. And that isn't complicated.
I have tried many books but I still don't have a good idea of how the mass of any object is determined relative to another.
The mass of a body is its resistance to change-in-motion. That's all it is. If we ignore friction, you can push a tricycle easily enough. A car is more difficult, because it has a greater mass. It exhibits a greater resistance to change in motion.
In The Science of Mechanics by Mach he says that two bodies isolated from all others are said to have same mass if they impart equal and opposite accelerations to each other. I think this assumes the third law. Also it is not that easy to measure their accelerations.
That's talking about active gravitational mass, which is really a measure of energy. A massless photon has a non-zero active gravitational mass. And it's a wave, not a body. An electron however is a body, and it's easy to measure its acceleration in response to a known force. That's how electron mass was measured.
Do we have any better and more useful definition of mass?
See above, and take a look at Einstein's E=mc² paper where he said "body" and "electron" on the same line. He said "the mass of a body is a measure of its energy-content". When you trap a massless photon in a mirror box, it increases the mass of that system. See Light is Heavy by van der Mark and (not the Nobel) 't Hooft. You know about pair production and electron diffraction and electron spin and the wave nature of matter. The electron is like the photon in the box, minus the box. Like it's in a box of its making. And electron-positron annihilation is like opening one box with another, whereupon each is a radiating body that loses mass. All of it.
Think of photon momentum as resistance to change-in-motion for a wave propagating on a linear path, think of electron mass as resistance to change-in-motion for a wave in a closed path, then think of your car as being made of electrons and similar things, because we can diffract protons and neutrons too. It isn't rocket science. It's simple.