When you pluck a string or hit a drum or sound a not on a flute, the instrument and the air in and around it vibrate and this vibration propagates as sound waves in the air to your hear drum.
When you hear an instrument being played, what you recognise as the note is the base frequency. 'C' corresponds to $261.6$ Hz and is the same for a piano or a guitar. But a 'C' played from a guitar, played from a piano or simply a $261.6$ Hz sounwave played from a computer speaker sound totally different. This is because of the overtones.
Let's look at the case of a string for a concrete example.
If you pluck the 'C' string on the guitar, you will hear the characteristic sound of it makes. This is because the string is vibrating at $261.6$ Hz, but also at a bunch of higher frequencies. These higher frequencies are called "overtones", and they are determined by the shape and build of the body of the guitar as well as the way you set the string in motion.
This is guitars with different shapes sound different. You can also try plucking or strumming the guitar string in various different places, and you will hear different tones of 'C'.
Overtones of the vibrating instrument are what makes each instrument (or voice, for that matter) sound different. The material, shape, and way the instrument is played all contribute to determine which overtones will be present.
The reason instruments sound more similar while holding a long note is that the overtones dissipate energy faster. Higher frequency vibrations generally lose energy quicker. So once the string is plucked or hit, the overtones start losing energy (thus lowering volume) faster than the base note, and after a while you only hear the base note.
TL;DR:
Instrument sound different because of the overtones they produce. These are higher frequencies than the note being played, and are determined by factors such as shape, material and way of playing the instrument and give the characteristic flavour to each instrument.