Can causality be violated? A common justification for prohibiting many unusual phenomena such as faster than light travel is that if they were possible, causality would be violated.
Let's define causality as:

You cannot change the past.

Meaning that at any given moment $t_1$, it is impossible to influence any event which took place at $t_0<t_1$.
Obviously, no one has ever heard of this being violated. Is there a reason why? Is it just because nobody has managed to build a time machine yet, or does something in physics expressly forbid causality from being violated? In other words, is causality a law of physics, and how much of physics would have to be rolled back and re-written if I built a time machine (it's probably easier to say what physics there would be left)?
There are some (in)famous paradoxes that arise if you are able to influence the past. There is probably no need to restate these for this question, unless these paradoxes per se are the justification for expecting causality to not be violated.

Question motivated by John Rennie's comment.
Some examples of how causality can be violated:


*

*A device which can travel faster than light

*A device which can travel into the past

*Newcomb's paradox
 A: To change the past you require a closed timelike curve. Stephen Hawking proved that closed timelike curves cannot be created in a finite system without using exotic matter. I think the proof was in his paper on the Chronology Protection Conjecture but I don't have access to the paper at the moment.
This far we have a reliable grasp on whether causality can be violated, but from here things get speculative. It seems likely that no infinite structures exist, if only because the universe hasn't existed for an infinite time (though if there was a Big Bounce we could be wrong about this). The big question is whether exotic matter exists.
The trouble is that there is no proof that exotic matter either exists or doesn't exist. It's like negative mass - there's nothing to stop you plugging a negative value for mass into the GR equations, but that doesn't mean it's a physically meaningful thing to do. We have never observed exotic matter, but that doesn't necessarily mean it doesn't exist.
The Chronology Protection Conjecture that I mentioned above is the closest we have to a mathematical approach to constraining causality violations, but it's just a conjecture and has not been proven - though it hasn't been disproven either. At the moment we simply don't understand the physics well enough to give a definitive answer.
A: 
Meaning that at any given moment $t_1$, it is impossible to influence any event which took place at $t_0<t_1$. Obviously, no one has ever heard of this being violated. Is there a reason why?

This is not really question about physics, but is related and interesting.
I assume by $t$ you mean time as measured by clock on the Earth. The only (hypothetical) way I know of how to change event that belongs to such past has been explained in sci-fi literature and movies. It is to travel back in time.
The easiest way to imagine that is that time travel is like space travel - you are transported to strange place which looks antique, but in your view, the time goes on as usual, i.e. you age and your watches go clock-wise as usual.
You can go on with life as usual. You can try to change some historical events so they would turn out differently than what you remember. In some you may succeed, some you may not. You may kill your grandfather or not. There is no direct paradox, because it is not clear how the time travel operates and whether you have to fulfill your destiny as everyone remembers it (then you would not succeed in preventing your birth) or you were transported to an entirely independent new "time-line" where you can prevent birth of a guy that is supposed to grow into another you.

does something in physics expressly forbid causality from being violated? 

Depends on what you mean by "causality" and how do your decide what is cause and what is effect? There are whole books written on this, but most physics does not depend in that - it is a problematic notion that is not really that important for mathematical formulation of physical laws.

other words, is causality a law of physics, and how much of physics would have to be rolled back and re-written if I built a time machine (it's probably easier to say what physics there would be left)?

Causality is not a law of physics, but a vague notion from daily life - a way people use to think about events to understand it.
If time machine was built, I do not think physics would have any problem with that. It would evolve and embrace it.
A: The problem with time is: its direction is subjective. Physical laws do not specify a distinction between "past" and "future" (if you look at quantum field theory, this is known as CPT symmetry).
That means: if you know the state of a system and you know the physical laws that govern it, you can calculate the future development with the same equations as the past of the system. You can even switch the sign of you time parameter and switch future and past!
What is usually meant by flow of time is the growth of entropy. It is perfectly normal, if a vase shatters on the floor. But if you would see shards jumping on the ground and assembling a vase by themselves, that would be very suspicious.
Imagine for an instance, what it would feel like to "travel backwards in time". Conversations would be paradoxical in themselves. If you meet someone else, it would be departure for them, they would have information about the whole conversation already. If you want to ask something, you would have to speak backwards, and when your partner answers after hearing the question, that would in your perspective take place before you asked the question. But if you know the answer already, why should you ask in the first place? So, "time travel" in itself is "paradoxical" already, you don't even need to kill your grandfather.
There is the notion of consistency to allow for time travel and prohibit these paradoxes. If you were to implement a means of FTL and then set out to kill your grandfather in the subjective past of earth, you would fail for some reason. Someone who is able and willing to kill his grandfather before he can sire a child is not consistent with a time travelling universe and simply would not exist.
A: It is not clear whether time travel is possible. Suggestions have been made for how a closed timelike curve could be constructed that have not yet been refuted, such as
http://arxiv.org/abs/gr-qc/0503077.
In addition, there have been attempts to solve the problems that result from the existence of closed timelike curves, such as David Deutsch's 1991 paper Quantum Mechanics Near Closed Timelike Lines. Short version: if you travel back into the past and change stuff you end up in a different universe in the Everett multiverse as described by quantum mechanics.
Newcomb's paradox violates good epistemology. The idea is that you can see one box with £1000 in it and there is another box which has either £1 million or £0. Whether there is £1 million in the box is determined by an agent that can somehow predict what choice you will make. It is not enough for the agent to predict your actions since you can learn from other people so he would have to be able to predict the actions of everybody else too. In order to get £1 million and a box and lots of information about you he has to interact with other people and so he will influence their actions. He can't predict exactly how he will interact with other people since that would involve predicting the growth of his own knowledge, which is impossible since if he knew today what he will know tomorrow he would already know it. So he can't predict your reactions perfectly because they are dependent on what he will do and he can't predict his own actions. So Newcomb's paradox is impossible.
