Intuitively, it means you imparted 100 J of energy to the object to bring it to that position. 

Your confusion arises from two sources (as far as I can tell):

1. You are conflating Work with Power
2. The definition of Work you are using. 

Work is **always** independent of time. It by definition, describes the difference in energy of a system between two states. Take your elephant and pen for example. We'll take the mass of the elephant, $m_e$, to be $1000 kg$ and the mass of the pen, $m_p$, to be $1 kg$. Now, as you stated, you exerted $1 N$ of force of each object for $100 m$. Finally, we'll assume their initial velocity was $0 m/s$. Using kinematics, we may conclude the following:

1. You applied a constant acceleration, $a = \frac{F}{m}$ of  to each object. The Elephant accelerated to $\frac{1N}{1000kg} = 10^{-3} m/s^2 $ and the pen accelerated to $\frac{1N}{1kg} = 1 m/s^2 $.
2. At $100m$, each object will have accelerated to $v_f = \sqrt{v_i ^2 + 2 ad} = \sqrt{2ad}$. The Elephant accelerated to $\sqrt{2 \cdot 10^{-3} m/s^2 \cdot 10^2 m} = 0.447 m/s$. The pen accelerated to $\sqrt{2 \cdot 1 m/s^2 \cdot 10^2 m} = 14.1 m/s$.
3. It took $t = \frac{v_f}{a}$ seconds to bring each object to 100m. For the Elephant it took $\frac{0.447 m/s}{10^{-3} m/s^2} = 447s $. For the Pen it took $\frac{14.1 m/s}{1 m/s^2} = 14.1 s $. 

Notice: Because the elephant was 100x more massive than the pen, it took significantly more time to push it 100m, and with a constant force it was traveling at one-tenth the velocity as the pen.

Now if we calculate the calculate the difference in kinetic energy to move the pen and elephant 100m:
$$T_e = \frac{1}{2} m_e v_{ef}^2 - \frac{1}{2} m_e v_{ei}^2 = 0.5 \cdot 1000kg \cdot (0.447 m/s^2)^2 - 0.5 \cdot 1000kg \cdot (0 m/s^2)^2 = 100 J$$
$$T_p = \frac{1}{2} m_e v_{ef}^2 - \frac{1}{2} m_e v_{ei}^2 = 0.5 \cdot 1 kg \cdot (14.1 m/s^2)^2 - 0.5 \cdot 1 kg \cdot (0 m/s^2)^2 = 100 J$$

You'll derive that the kinetic energy is identical. According to the Work-Energy theorem, the difference in kinetic energy of the system is equal to the total work done on the system. 

Still confused? Take a look at the power used to move the objects:

$$P = \frac{W}{t} = \frac{Fd}{t} = Fv $$

You'll find that the power used to move the elephant was 0.224 Watts and the power used to move the pen was 7 Watts. You still expended the same amount of energy for either scenario; but, you expended energy at a faster rate pushing the pen, than pushing the elephant. This may seam counter-intuitive; but, remember, you exerted 1 N of force on a **friction-less surface**. On surfaces with friction, we must exert enough force to overcome [static friction][1] *before the object will move*.

EDIT: corrected erroneous calculations. 


  [1]: http://hyperphysics.phy-astr.gsu.edu/hbase/frict2.html
  [2]: https://en.wikipedia.org/wiki/Work_(physics)
  [3]: https://en.wikipedia.org/wiki/Work_(physics)#Mathematical_calculation