In cosmology, "flat" doesn't mean the opposite of "rough." It means the opposite of "curved in a global sense." The surface of a sphere is not flat, even if it is smooth, as the surface everywhere has positive curvature. A saddle has everywhere negative curvature. An uncurved plane is flat in the cosmological sense, even if it has some bumps and ripples on it.
As you can (and should) read on Wikipedia, very basic cosmology just assumes a few different constituents of the universe - normal matter, dark matter, radiation, dark energy - uniformly distributed everywhere. You can then ask
- What is the overall curvature of the universe with this composition, according to general relativity?
- How do the relative amounts of different components (including curvature) evolve over time, again according to general relativity.
It is easily shown that flatness comes from a fine balance between the components, and, what's more, any slight deviation from perfect flatness will grow over time. If the universe started off slightly positively curved, it would be enormously positively curved today, and the same holds for negative curvature.
The problem is that we observe the present-day universe to be flat to within the precision of our measurements. Inflation is a proposed mechanism by which the universe was flattened out very well early on, so that curvature will not have grown out of control by the present era.
Note that in some sense this only pushes the question further back. After all, given a fixed amount of smoothing caused by inflation, you can always choose pre-inflation initial conditions to be curved enough such that inflation doesn't fix it. Ideas like this have led some cosmologists to question how well inflation really solves things (PDF here). The ideas behind inflation have changed a lot over the last several decades, and they are still evolving.