Could be one of two things (or both):

1. Cool air being forced down the chimney taking smoke with it - that is, we know that cold air sinks (and hot air rises), and so a column of cold air sinking can force smoke back down into the room.

2. It could also be simply a matter of winds blowing downward pushing smoke back inside.

Could be one of two things:

1. Cool air being forced down the chimney taking smoke with it - that is, we know that cold air sinks (and hot air rises), and so a column of cold air sinking can force smoke back down into the room.

2. It could also be simply a matter of winds blowing downward pushing smoke back inside.