Is there a way to "build" special relativity by using causality as one of the axioms? Or is it just a consequence of a fact that the speed of light is same in all reference frames?
Not quite. Galilean relativity, which has absolute time, is definitely causal; you can't have future events influence past ones because all observers agree on all times.
However, it is interesting how much relativity you can get with incomplete axioms. There's a derivation on page 38 here that shows you can get the Lorentz transformations, but with a general $v$ in place of $c$, using just the postulate that inertial frames are equivalent. Then adding in the 'speed of light is same' axiom sets $v = c$. Like your causality thing, this doesn't rule out Galilean relativity because that's just the limit $v \rightarrow \infty$.
Given that, I'm not sure if just a "causality" axiom is enough to get equally far. Maybe you could argue that causality really is the same thing as "all inertial frames are equivalent", but I'm not sure about that.
Based on this post:
and the content within, the issue of causality seems to me to be a philosophical as much as a physical question. So I would guess there is no clearcut answer to your question.
Have a read, see what you think, and see will you end up as confused as everybody else seems to be regarding the issue.