What is a joule?
It is unit of work, and work is a scalar quantity defined as
$$W = \int \vec{F} \cdot d\vec{s}$$
where $\vec{F}$ is force and $\vec{s}$ is displacement. Note that dot between force and displacement represents scalar product, which means that only force component parallel to displacement does work, while the perpendicular component does no work. From this definition we can conclude that unit of work is $\text{Nm}$ which is assigned a special unit called Joule: $1 \text{ J} = 1 \text{ Nm}$.
But now lets replace the ball bearing with a bowling ball. If I push on it with a force of 1 newton until it moves 1 metre, it will accelerate much slower, it will take much longer to move 1M, so I'm pushing on it with a force of 1N for longer, so I feel like I have done more work moving it compared to the ball bearing.
Everything would have made sense if you knew about the work-energy theorem
$$\Delta K = W$$
The above theorem was derived directly from the second Newton's law of motion and from the definition of work. What it says is that change of kinetic energy equals total work done on an object. Since kinetic energy is defined as $K = \frac{1}{2} m v^2$ the above equation can also be written as
$$\frac{1}{2} m (v_f^2 - v_i^2) = W$$
where $v_f$ and $v_i$ are final and initial velocities, respectively. For the same amount of work done on two objects with different mass, the speed of the heavier object will change less than speed of the lighter object. It takes more work to accelerate heavier object to the same speed difference compared to lighter object, which is kind of intuitive.
it will accelerate much slower, it will take much longer to move 1M
The common misconception is to compare work done by two forces by looking for how long they were acting. Work by definition is not about time, it is about displacement. Integral of force over time is impulse which is related to the impulse-momentum theorem
$$\vec{J} = \int \vec{F} dt \qquad \text{and} \qquad \Delta \vec{p} = \vec{J}$$
which is (slightly) different concept to work, but equally useful.
I will conclude this with a paragraph about abstract concept of energy:
"It is important to realize that in physics today, we have no knowledge of what energy is. We do not have a picture that energy comes in little blobs of a definite amount. It is not that way. However, there are formulas for calculating some numerical quantity, and when we add it all together it gives "28" -- always the same number. It is an abstract thing in that it does not tell us the mechanism or the reasons for the various formulas.
From "The Feynman Lectures on Physics", Vol 1., 4-1 What is energy?