First, you need a deterministic dynamical system. By deterministic it means that the state of the system is univocally determined at each time, ie. at each time you have one and only one possible state. In the counterpart are stochastic systems where, instead, the state of the system is determined by a distribution of possible states and is this distribution that evolves in time.
Secondly, the main idea behind chaos is that the evolution of the system is hard to predict in the following sense. The evolution of the system is usually determined by a (set of) equations of evolution. These equations usually admit a family of solutions and you can specify one and only one of them choosing the initial (plus some times boundary) condition(s). In general, a variation in the initial condition will give you different solutions or trajectories for the evolution of the system. In chaotic systems a small variation of the initial condition leads to a trajectory which is very different from the original, but both get infinitely close (or "mixed") to the same attractor. This has the implication that if you have an "error" in the initial condition then your evolution will be very different from the initial condition without the error, but as both approaches the same attractor both trajectories becomes almost indistinguishable.
Usually the main idea behind a chaotic system is that two trajectories deviate one of the other exponentially fast. But this may happen also with non chaotic systems. Consider for instance an exponential growth. It is not periodic, and it is sensitive to initial conditions, but for sure it is not chaotic. More precisely, if $y(x,t)$ is an exponential growth with initial condition $x$ and running on time $t$, then
$y(x+\delta x,t) = e^{(x+\delta x)t} = e^{xt}.e^{(\delta x)t} = y(x,t).e^{(\delta x)t}$
indicating that the perturbed solution (left hand side) deviates exponentially fast from the non perturbed one (right hand side). The main difference is that in chaotic system (chaotic) trajectories are confined to live in a bounded space, so they become "mixed". So one possible formal definition of a chaotic (assumed deterministic) system is: a system where trajectories with different initial conditions deviates exponentially in time, but the trajectories are restricted to live in a bounded space. There exist some weird systems where these conditions can be violated but those are not the typical cases. In the end, the important property is "mixing", ie. the perturbed trajectory can become arbitrarily close and far away (respecting the space boundaries if present) from the original one.
Edit: Thanks for the edit GuySoft. But there is a mistake in your correction. Is not a definition of chaotic system that two trajectories with different initial conditions deviate exponentially. It is just a property of them. You need also the extra property that trajectories must be bounded.