Is time dilation due to the travel time of light? I'm trying to think about special relativity without "spoiling" it by looking up the answer; I hope someone can offer some insight - or at least tell me I'm wrong.  
Suppose I have an ordinary clock in front of me and I push it back with my hands.  The force applied to the clock causes it to retreat away from me and after the push, it will travel away with uniform velocity.  Suppose further, I can always see the clock clearly no matter how far away it is.  Since the speed of light is constant, the light coming from the clock must travel a longer distance to reach my eye as it moves away.  This would make time appear to slow down?  If, on the other hand, the clock is moving towards me, the distance the light must travel to reach my eye becomes shorter and shorter, thus time would appear to speed up?
 A: Relativity is not needed: If you replace the clock with a strong laser which fires a ray of light every second with an atomic clock, you know that the ticks will not slow down because every tick will be followed by another after a second from the laser's perspecive . But how can you arrange that with the fact that the light move further and further away ?
Simple: As the clock is moving away from you with constant speed, every interval the distance will increase by an amount. This additional amount needs to be traversed by the light ray. Lets say the clock moves with 1 m/s. Then your measured time interval between the ticks increases from 1s to 1+(1/300 000 000) s, which is imperceptible. But this time accumulates and after 300 000 000 ticks, you have finally reached a 1s delay which is identical with the distance the clock is now away from you: 300 000 000 m.
So your measured time between the ticks does not slow down with distance, but increases from the start of releasing the clock and remains constant then.
A: Analyzing one moving clock from the perspective of one stationary person will be inadequate to derive special relativity from.  With just that set-up, you aren't actually using the key fact that the speed of light is the same for all observers – all you're actually using is just the fact that the speed of light is finite.  With just taking into account that the speed of light is finite, all you'll arrive at is the non-relativistic Doppler effect, which is different from time dilation.
A: In special relativity there is a distinction between 'experiencing events' and the concept of an observer:

Speaking of an observer in special relativity is not specifically
  hypothesizing an individual person who is experiencing events, but
  rather it is a particular mathematical context which objects and
  events are to be evaluated from

In particular the observed clock ticks at the same rate, whether the clock moves away or towards you, since you don't measure the ticks from your point of view, but rather as they appear relative to imaginary stationary clocks that you placed in advance in the moving clock's path.
Einstein explains this beautifully in his popular book Relativity; the special and general theory
And Feynman also has a great exposition in http://www.feynmanlectures.caltech.edu/I_15.html
A: No matter in which direction the clock moves (away or towards you) , the time will slow down in the clock. Time will always dilate as the clock moves faster and faster, but will be apparent to a human eye only once it reaches speeds close to the light speed. This is not taking into effect the doppler shift which is merely the increase/decrease in the frequency of light.
A: A photon is fired across the width of a space ship a distance of one unit “ud”. The space ships velocity v relative to a fixed observer is 0.8*c
The observer sees the photon travel a distance of cT and knowing the width to be one unit of distance calculates that
1ud = T*(c^2-v^2)^0.5
1ud =  Tc*(1-(v/c)^2)^0.5
1ud = 0.6 Tc
A photon is now fired forward in the space ship and in time T the observer sees it travel a distance Tc. The space ship has moved forward a distance of Tv so the photon is seen to be in a position fwd of its origin by
Tc -  Tv
1ud/0.6 – 0.8ud/0.6 = 1/3 ud
As the laterally moving photon travels a unit of distance, it appears that the longitudinal dimensions of the space craft has been reduced to one third of its length.
Now a photon is fired aft and the observer sees it travel a distance Tc and relative to the space ship a distance of
Tc  + Tv
1ud/0.6  + 0.8ud/0.6  = 3ud
As the laterally moving photon travels a unit of distance, it appears that the longitudinal dimensions of the space craft has been increased to three times its length.
There is another way of looking at this issue.  The difference in speed between the space ship and a photon travelling in the same direction (but not inside the ship) is 0.2*c. Our observer sees a photon inside the ship moving forward and says its speed relative to his position is c.  The ship has a set of longitudinal markers unit distance apart. Our observer knows that the photons apparent speed relative to the ship is 0.2*c but a photon within the ship is travelling at c relative to the ship . In order to rationalize the observations t he observer concludes that the markers are 0.2 units apart from his perspective.
Now there is photon travelling in the opposite direction to the ship and the relative speed as seen by the observer is 1.8*c and therefore the markers are 1.8 units apart
A: Your statement:
"Since the speed of light is constant, the light coming from the clock must travel a longer distance to reach my eye as it moves away. This would make time appear to slow down? If, on the other hand, the clock is moving towards me, the distance the light must travel to reach my eye becomes shorter and shorter, thus time would appear to speed up?"
is correct.  But this is due to Doppler effect.  Doppler effect exists even for sound: a sound source moving away from you sounds lower pitch, a sound source moving toward you sounds higher pitch.
The exact formula for how fast or slow time appears depends on the fact that speed of light is constant (hence using the premise of special relativity), but qualitatively speaking, this thought experiment itself does not really present the essence of special relativity.  (Qualitatively speaking, this experiment is NOT a demonstration of the famous time dilation of special relativity.) 
