So, I know that changing magnetic field produces a non electrostatic electric field which goes around in circles. My question is how do we know they are circular in nature? If I'll put a square wire in changing magnetic field then the current must go in a square and so should the electric field. Isn't it right?
The electric field would shoot out radially from the wire. The magnetic field would loop around in circles. The E and B field must always be orthogonal; the B field runs tangent to the circumference of the wire while the E field is normal to it's surface. How we know they are orthogonal is how we know everything else; we observe it, over, and over, and over again. Mathematically we then represent it with the cross product.
Now as for the whole square thing, how the [individual] E and B vectors generated from a single point on the wire add onto one another to create the TOTAL E and B fields for a given geometry... that can get messy. Symmetry helps. In the case of square I don't recall off the top of my head though.
In absence of other elements, the shape adopted will be circular due to the simmetry. Else there would be "priviledged" or "favoured" directions, but that does not make sense because rotating the system should not make any change, as the source elements are indistinguishable from the previous situation. Once you introduce a "distubing element" you are changing the boundary conditions and the fields adapt to it.