Why does brain control opposite halves of the body?

This picture shows three diagrams.

The circles show the retina, the lines are optic tracts and curved lines resemble the cortex

The third diagram in the picture shows what may have been the arrangement- the right cortex receives visual signals from right eye. But the images on the nasal side will be seen on the temporal retinas of both eyes, which will be seen as double. So the arrangement is not feasible.

Now we come to the first diagram, this is what actually is-

The nasal fibres (carrying impulses of temporal field of vision) cross over. 

And since right cortex receives impulses from left side, it sends motor impulses also to the left side

Now here is my question-

Why don't the temporal fibres cross?

If that would happen, the same half of brain will control same side of the body! Human anatomy could have been so much simple 

If you get answer to this question-

Plz post it as comments 

Thanking you